diff --git a/CONTRIBUTING.html b/CONTRIBUTING.html new file mode 100644 index 000000000..9c40a62c3 --- /dev/null +++ b/CONTRIBUTING.html @@ -0,0 +1,351 @@ + + + + + + + + + Contributing — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Contributing

+

Some tips and guidelines for developers hacking on BuildStream

+
+

Filing issues

+

If you are experiencing an issue with BuildStream, or would like to submit a patch +to fix an issue, then you should first search the list of open issues +to see if the issue is already filed, and open an issue +if no issue already exists.

+
+
+

Fixing bugs

+

Before fixing a bug, it is preferred that an issue be filed +first in order to better document the defect, however this need not be followed to the +letter for minor fixes.

+

Patches which fix bugs should always come with a regression test.

+
+
+

Adding new features

+

Feature additions should be proposed on the mailing list +before being considered for inclusion. To save time and avoid any frustration, +we strongly recommend proposing your new feature in advance of commencing work.

+

Once consensus has been reached on the mailing list, then the proposing +party should file an issue to track the +work.

+

New features must be well documented and tested in our test suite.

+

It is expected that the individual submitting the work take ownership +of their feature within BuildStream for a reasonable timeframe of at least +one release cycle after their work has landed on the master branch. This is +to say that the submitter is expected to address and fix any side effects, +bugs or regressions which may have fell through the cracks in the review +process, giving us a reasonable timeframe for identifying these.

+
+
+

Submitting patches

+

Submitting patches works in the regular GitHub workflow of submitting +pull requests.

+
+

Branch names

+

If you are an apache member with access to the main repository, and are +submitting a pull request for a branch within the main repository, then +please be careful to use an identifiable branch name.

+

Branch names for pull requests should be prefixed with the submitter’s +name or nickname, followed by a forward slash, and then a descriptive +name. e.g.:

+
username/fix-that-bug
+
+
+

This allows us to more easily identify which branch does what and +belongs to whom, especially so that we can effectively cleanup stale +branches in the upstream repository over time.

+
+
+

Pull requests

+

Once you have created a local branch, you can push it to the upstream +BuildStream repository using the command line:

+
git push origin username/fix-that-bug
+
+
+

GitHub will respond to this with a message and a link to allow you to create +a new merge request. You can also create a pull request using the GitHub UI.

+

You may open pull requests for the branches you create before you are ready +to have them reviewed and considered for inclusion if you like. Until your merge +request is ready for review, the pull request title must be prefixed with the +WIP: identifier.

+

Consider marking a pull request as WIP again if you are taking a while to +address a review point. This signals that the next action is on you, and it +won’t appear in a reviewer’s search for non-WIP merge requests to review.

+
+
+

Organized commits

+

Submitted branches must not contain a history of the work done in the +feature branch. For example, if you had to change your approach, or +have a later commit which fixes something in a previous commit on your +branch, we do not want to include the history of how you came up with +your patch in the upstream master branch.

+

Please use git’s interactive rebase feature in order to compose a clean +patch series suitable for submission upstream.

+

Every commit in series should pass the test suite, this is very important +for tracking down regressions and performing git bisections in the future.

+

We prefer that documentation changes be submitted in separate commits from +the code changes which they document, and newly added test cases are also +preferred in separate commits.

+

If a commit in your branch modifies behavior such that a test must also +be changed to match the new behavior, then the tests should be updated +with the same commit, so that every commit passes its own tests.

+

These principles apply whenever a branch is non-WIP. So for example, don’t push +‘fixup!’ commits when addressing review comments, instead amend the commits +directly before pushing.

+
+
+

Commit messages

+

Commit messages must be formatted with a brief summary line, followed by +an empty line and then a free form detailed description of the change.

+

The summary line must start with what changed, followed by a colon and +a very brief description of the change.

+

If the commit fixes an issue, or is related to an issue; then the issue +number must be referenced in the commit message.

+

Example:

+
element.py: Added the frobnicator so that foos are properly frobbed.
+
+The new frobnicator frobnicates foos all the way throughout
+the element. Elements that are not properly frobnicated raise
+an error to inform the user of invalid frobnication rules.
+
+Fixes #123
+
+
+

Note that the ‘why’ of a change is as important as the ‘what’.

+

When reviewing this, folks can suggest better alternatives when they know the +‘why’. Perhaps there are other ways to avoid an error when things are not +frobnicated.

+

When folks modify this code, there may be uncertainty around whether the foos +should always be frobnicated. The comments, the commit message, and issue #123 +should shed some light on that.

+

In the case that you have a commit which necessarily modifies multiple +components, then the summary line should still mention generally what +changed (if possible), followed by a colon and a brief summary.

+

In this case the free form detailed description of the change should +contain a bullet list describing what was changed in each component +separately.

+

Example:

+
artifact cache: Fixed automatic expiry in the local cache
+
+  o _artifactcache/artifactcache.py: Updated the API contract
+    of ArtifactCache.remove() so that something detailed is
+    explained here.
+
+  o _artifactcache/cascache.py: Adhere to the new API contract
+    dictated by the abstract ArtifactCache class.
+
+  o tests/artifactcache/expiry.py: Modified test expectations to
+    match the new behavior.
+
+This is a part of #123
+
+
+
+
+
+

Committer access

+

Committers in the BuildStream project are those folks to whom the right to +directly commit changes to our version controlled resources has been granted.

+

While every contribution is valued regardless of its source, not every person +who contributes code to the project will earn commit access. +The COMMITTERS file lists all committers.

+
+

How commit access is granted

+

After someone has successfully contributed a few non-trivial patches, some full +committer, usually whoever has reviewed and applied the most patches from that +contributor, proposes them for commit access. This proposal is sent only to the +other full committers - the ensuing discussion is private, so that everyone can +feel comfortable speaking their minds. Assuming there are no objections, the +contributor is granted commit access. The decision is made by consensus; there +are no formal rules governing the procedure, though generally if someone strongly +objects the access is not offered, or is offered on a provisional basis.

+

This of course relies on contributors being responsive and showing willingness +to address any problems that may arise after landing patches. However, the primary +criterion for commit access is good judgement.

+

You do not have to be a technical wizard or demonstrate deep knowledge of the +entire codebase to become a committer. You just need to know what you don’t +know. Non-code contributions are just as valuable in the path to commit access. +If your patches adhere to the guidelines in this file, adhere to all the usual +unquantifiable rules of coding (code should be readable, robust, maintainable, etc.), +and respect the Hippocratic Principle of “first, do no harm”, then you will probably +get commit access pretty quickly. The size, complexity, and quantity of your patches +do not matter as much as the degree of care you show in avoiding bugs and minimizing +unnecessary impact on the rest of the code. Many full committers are people who have +not made major code contributions, but rather lots of small, clean fixes, each of +which was an unambiguous improvement to the code. (Of course, this does not mean the +project needs a bunch of very trivial patches whose only purpose is to gain commit +access; knowing what’s worth a patch post and what’s not is part of showing good +judgement.)

+
+
+
+

Windows CI

+

The infrastructure for running the CI against Windows is different from the usual +runners, due to a combination of licensing technicalities and differing +containerisation support.

+

The scripts used to generate a CI runner can be found at +https://gitlab.com/BuildStream/windows-startup-script. +The wsl branch can be used to generate a runner for WSL, and the win32 branch +can be used to generate a native-windows runner.

+
+
+

Further information

+
+ +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/_images/arch-datamodel-context.svg b/_images/arch-datamodel-context.svg new file mode 100644 index 000000000..b22fbf33e --- /dev/null +++ b/_images/arch-datamodel-context.svg @@ -0,0 +1,211 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Command Line ArgumentsArguments specified on the command line have the final say for any user preference.User ConfigurationThe user configuration file overrides any of the previous settings.Project RecommendationsWhile we try to keep project data and user preferences as separate as possible, in some cases it is convenient for the project to suggest a value, such as which artifact server to use.Default ConfigurationBuildStream ships an internal YAML file with the same structure as the user configuration file, this file specifies all of the default configurations. + + + + + + + + + + + + + Default Configuration + + + + + + + + + + + + + Project Recommend + + + + + + + + + + + + + User Configuration + + + + + + + + + + + + + Command Line Arguments + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/arch-datamodel-element-composition.svg b/_images/arch-datamodel-element-composition.svg new file mode 100644 index 000000000..985fe7213 --- /dev/null +++ b/_images/arch-datamodel-element-composition.svg @@ -0,0 +1,463 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BuildStream Core DataPublic Domain Data + + + + + + + + + + + + + Core Defaults + + + + + + + + + + + + + Element Declaration + + + + + + + + + + + + + Project Configuration + + + + + + + + + + + + + Element Defaults + + + + + + + + + + + + + Project Overrides + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Element Configuration Data + + + + + + + + + + + + + Element Declaration + + + + + + + + + + + + + Element Defaults + + + + + + + + + + + + + Project Overrides + + + + + + + + + + + + + + + + + + + + Element Configuration Data is composited in the same way as Core Data and Public Data, except that it is not understood by the project configuration so there are no defaults. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Composited the first time that a Element of a given “kind” is instantiated.This is cached on the given Element’s class data for later reuse. + + + + + + Composited every time against the cached class data below every time an Element is instantiated. + + + + + + + \ No newline at end of file diff --git a/_images/arch-datamodel-element.svg b/_images/arch-datamodel-element.svg new file mode 100644 index 000000000..ab75414da --- /dev/null +++ b/_images/arch-datamodel-element.svg @@ -0,0 +1,183 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BuildStream Core DataCore data is understood by the BuildStream core. This includes the list of source declarations which are components of the Element.Element Configuration DataElement Configuration Data is defined by Element plugin classes. This data is internal to a given plugin.Public Domain DataPublic Domain Data is free form and not validated, this provides a way to declare data that is intended to be read back and processed by reverse dependencies. + + + + + + Elementkind: autotoolsdescription: Optional description about foodepends:- gtk+.bst- clutter.bstsources:- <source declarations here>config: configure-commands: (<): - cp %{datadir}/automake-*/config.{sub,guess} .public: bst: integration-commands: - update-flying-pony-cache -f ${datadir}/ponies + + + + + + + \ No newline at end of file diff --git a/_images/arch-datamodel-source-composition.svg b/_images/arch-datamodel-source-composition.svg new file mode 100644 index 000000000..60de23620 --- /dev/null +++ b/_images/arch-datamodel-source-composition.svg @@ -0,0 +1,255 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BuildStream Core DataSource Configuration Data + + + + + + + + + + + + + Source Declaration + + + + + + + + + + + + + Project Overrides + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Composited the first time that a Source of a given “kind” is instantiated.This is cached on the given Source’s class data for later reuse.Note: In the case of Sources, defaults are provided in code. + + + + + + Composited every time against the cached class data below every time an Source is instantiated. + + + + + + + + + + + + + Source Defaults + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/arch-datamodel-source.svg b/_images/arch-datamodel-source.svg new file mode 100644 index 000000000..55e303311 --- /dev/null +++ b/_images/arch-datamodel-source.svg @@ -0,0 +1,167 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BuildStream Core DataCore data understood by BuildStream.Source Configuration DataSource Configuration Data is defined by Source plugin classes. + + + + + + Sourcekind: gitdirectory: sub/diruri: upstream:foo.gittrack: masterref: bbf775301a08b9a578ef7f647bc35fe13e816241 + + + + + + + \ No newline at end of file diff --git a/_images/arch-dependency-model-build.svg b/_images/arch-dependency-model-build.svg new file mode 100644 index 000000000..67bc5b4d2 --- /dev/null +++ b/_images/arch-dependency-model-build.svg @@ -0,0 +1,267 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Build scope for the Application element + + + + + + + + + + + + + Application + + + + + + + + + + + + + ServiceRequired at runtime + + + + + + + + + + + + + CompilerRequired at build time + + + + + + + + + + + + + Base RuntimeRequired at all times + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Service DefinitionXML files needed to build the Service + + + + + + + + + + + + + + + + + + + + BootstrapSelf Hosting Compiler + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/arch-dependency-model-runtime.svg b/_images/arch-dependency-model-runtime.svg new file mode 100644 index 000000000..dc758c844 --- /dev/null +++ b/_images/arch-dependency-model-runtime.svg @@ -0,0 +1,266 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Runtime scope for the Application element + + + + + + + + + + + + + Application + + + + + + + + + + + + + ServiceRequired at runtime + + + + + + + + + + + + + CompilerRequired at build time + + + + + + + + + + + + + Base RuntimeRequired at all times + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Service DefinitionXML files needed to build the Service + + + + + + + + + + + + + + + + + + + + BootstrapSelf Hosting Compiler + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/arch-dependency-model.svg b/_images/arch-dependency-model.svg new file mode 100644 index 000000000..3c755b023 --- /dev/null +++ b/_images/arch-dependency-model.svg @@ -0,0 +1,299 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Application + + + + + + + + + + + + + ServiceRequired at runtime + + + + + + + + + + + + + CompilerRequired at build time + + + + + + + + + + + + + Base RuntimeRequired at all times + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + LegendBuild DependencyRuntime DependencyBuild & Runtime Dependency + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Service DefinitionXML files needed to build the Service + + + + + + + + + + + + + + + + + + + + BootstrapSelf Hosting Compiler + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/arch-overview.svg b/_images/arch-overview.svg new file mode 100644 index 000000000..738a6a800 --- /dev/null +++ b/_images/arch-overview.svg @@ -0,0 +1,2053 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Frontend + + + + + + + + + + + + + Main Entry Point (cli.py)Implements command line interface + + + + + + + + + + + + + Main Application State (App)Initializes the “Stream”, handles logging and user interactions + + + + + + + + + + + + + + + + + + + + Loggerwidget.py + + + + + + + + + + + + + Status Barstatus.py + + + + + + + + + + + + + Completionscomplete.py + + + + + + + + + + + + + Color profilesprofile.py + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Core, Frontend facing API + + + + + + + + + + + + + ContextUser configuration, CLI arguments + + + + + + + + + + + + + + + + + + + + ProjectLoaded project.conf configuration + + + + + + StreamLoading of the project, implementation of all main user facing commands in the CLI + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Core Internals + + + + + + + + + + + + + PlatformAbstract class providing platform specific routines. + + + + + + The Frontend implements the main command line interface, status bar, master log formatting, and any interactivity with the user. This is cleanly separated from the core BuildStream application. + + + + + + + + + + + + + Scheduler(Scheduling of element processing jobs)The Scheduler is in charge of processing elements in job queues. + + + + + + + + + + + + + Job(Abstract tasks) + + + + + + + + + + + + + Queue(Abstract job queues) + + + + + + + + + + + + + Loader(Load the elements in a project)The loader loads elements and sources from the project directory, transforming them into data structures ready for Element and Source instantiation + + + + + + + + + + + + + LoadElement, Dependency(Loader data model) + + + + + + The Frontend creates the toplevel Project, Context and Stream, and receives Elements and Sources by way of loading the Stream. + + + + + + These represent the main active internal components which sit below the frontend facing business logic in terms of module stacking order. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + OptionsLoads and resolves project options, and processes conditional statements + + + + + + + + + + + + + VariablesResolves a collection of variables, and performs substitutions on strings containing variables + + + + + + + + + + + + + IncludesProcesses include statements and composites the resulting dictionaries + + + + + + YAML Parsing Utilities + + + + + + + + + + + + + YAMLLow level utilities for parsing YAML, storing the provenance of loaded values for error reporting, composition of dictionaries and validation of loaded values + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + utils.pyVarious miscellaneous utilities, includes the directory staging mechanics + + + + + + + + + + + + + signals.pyUtilities and context managers for dealing with UNIX signal handling + + + + + + + + + + + + + utils.pyVarious miscellaneous utilities, includes the directory staging mechanics + + + + + + + + + + + + + WorkspacesHelper object for (de)serialization of the workspace configuration + + + + + + Low Level Subsystems + + + + + + + + + + + + + Element, Source, and PluginData Model + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CAS Server + + + + + + + + + + + + + CasServerContent Addressable Storage Server + + + + + + The CAS Server only needs to interact with Core Internals and lower levels of the stack. + + + + + + + + + + + + + CasCacheContent Addressable Storage implementation, used by the artifact cache and CAS server + + + + + + + + + + + + + FUSE / SafeHardlinksThe Fuse layer implements a copy on write hardlink experience for the Sandbox implementations + + + + + + + + + + + + + SourceFactory / ElementFactoryLow level factory for instantiating Source and Element plugins, one plugin namespace per project + + + + + + + + + + + + + ArtifactCacheArtifact Storage Module + + + + + + + + + + + + + SandboxAbstract Class for running commands in the isolated build environment + + + + + + + + \ No newline at end of file diff --git a/_images/arch-program-flow.svg b/_images/arch-program-flow.svg new file mode 100644 index 000000000..ba196a90d --- /dev/null +++ b/_images/arch-program-flow.svg @@ -0,0 +1,1819 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Parse Command Line + + + + + + + + + + + + + + + + + + + + Load Context + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Enter Application Context + + + + + + + + + + + + + + + + + + + + Load Platform + + + + + + + + + + + + + + + + + + + + ArtifactCache Preflight + + + + + + + + + + + + + + + + + + + + Initialize Logging + + + + + + + + + + + + + + + + + + + + + + + + + + + Exit with error + + + + + + + + + + + + + Load Toplevel Project + + + + + + + + + + + + + + + + + + + + Initialize Stream + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Initialize Application + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Run Stream API + + + + + + + + + + + + + Load Elements + + + + + + + + + + + + + + + + + + + + Resolve Keys + + + + + + + + + + + + + + + + + + + + Resolve Cached State + + + + + + + + + + + + + + + + + + + + Construct Scheduler Queues + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + EventSwitch + + + + + + + + + + + + + + + + + + + + Log Line + + + + + + + + + + + + + + + + + + + + Logging Event + + + + + + Interrupt or Error Event + + + + + + + + App isInteractive + + + + + + + + + + + + + Run SchedulerRun the event loop, and process the elements until they have passed through all of the queues, or until there is a terminate condition. + + + + + + + + + + + + + Interactive + + + + + + + + + + + + + Pause Scheduler + + + + + + + + Prompt User + + + + + + + + + + + + + + + + + + + + Resume Scheduler + + + + + + + + + + + + + Set Terminate Condition + + + + + + + + + + + + + + + + + + + + Non Interactive + + + + + + Continue + + + + + + + + + + + + + Set Terminate Condition + + + + + + + + + + + + + Terminate + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Log Session Summary + + + + + + + + + + + + + Clean Exit + + + + + + + + + + + + + Exception + + + + + + Event + + + + + + + \ No newline at end of file diff --git a/_images/arch-remote-execution.svg b/_images/arch-remote-execution.svg new file mode 100644 index 000000000..e55bebb2d --- /dev/null +++ b/_images/arch-remote-execution.svg @@ -0,0 +1,499 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Remote Execution service + + + + + + + + + + + + + Local artifact cache + + + + + + push + + + + + + + + + + + + + + REAPI + + + + + + + + Remote artifact cache 1 + + + + + + + + + + + + + + + + + + + + pull + + + + + + + + Remote artifact cache 2 + + + + + + push + + + + + + + + + + + + + + + + + + + + pull + + + + + + + + + + + + + + Send (initial environments, commands) + + + + + + + Retrieve (finished environments, logs) + + + + + + + + Cache manager(cascache.py) + + + + + + + + Sandbox.run(commands, flags, environment) + + + + + + + + Remote execution cache + + + + + + + + REAPI-compatible build service + + + + + + BuildStream + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Remote execution service implementation + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/arch-scheduler-job.svg b/_images/arch-scheduler-job.svg new file mode 100644 index 000000000..9a50135d8 --- /dev/null +++ b/_images/arch-scheduler-job.svg @@ -0,0 +1,750 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Setup signal handlers + + + + + + + + + + + + + Block signals + + + + + + + + + + + + + fork() + + + + + + + + + + + + + Unblock signals + + + + + + + + + + + + + + + + + + + + + + + + + + + Main BuildStream Process + + + + + + Job Process + + + + + + + + + + + + + Initialize subprocess logging redirection + + + + + + + + + + + + + + + + + + + + Unblock selected signals (ready) + + + + + + + + + + + + + + + + + + + + Run the job payload + + + + + + + + + + + + + + + + + + + + Log messages + + + + + + + + + + + + + + + + + + + Start New Job + + + + + + + + + + + + Receive IPC Messages + + + + + + + + + + + + + Report job result and any other state + + + + + + + + + + + + + Collect results on job instance + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Collect exit status + + + + + + + + + + + + + Exit with symbolic exit status + + + + + + + + + + + + Interruptible Job Harness + + + + + + + + + + + + + Inform the job owner that a job completed, hand over the result + + + + + + + + + + + + + + + + + + + + + + + + + + Job ControlSuspend (SIGTSTP)Resume (SIGCONT)Terminate (SIGTERM)Kill (SIGKILL) + + + + + + + + + + + + + Job Exited + + + + + + + + + + + + + This process is a fork() of the main process without execve().This means it has access to the main process data model at the time of the fork(), any modifications made after this point are copy-on-write, and any side effects of the job need to be manually propagated back to the main process through the IPC. + + + + + + + \ No newline at end of file diff --git a/_images/arch-scheduler-queue-ports.svg b/_images/arch-scheduler-queue-ports.svg new file mode 100644 index 000000000..2466551bf --- /dev/null +++ b/_images/arch-scheduler-queue-ports.svg @@ -0,0 +1,318 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Processed + + + + + + + + + + + + + Failed + + + + + + + + + + + + + Skipped + + + + + + + + + + + + + + + + + + + Ready + + + + + + + + + + + + + Skip + + + + + + Queue implementations report a “QueueStatus” for all of the elements which are in the input queue at all times.Skip elements go directly to the output queue without processing, and the scheduler only ever processes the Ready elements. + + + + + + + + + + + + + Wait + + + + + + + + + + + + + Elements In + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Element Processing + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + After elements are either processed or skipped, they move to the Queue’s output queue where the scheduler can pick them up and and move them along to the next Queue.Elements which have passed through a Queue are also kept in status lists for bookkeeping purposes. + + + + + + Input Queue + + + + + + Output Queue + + + + + + + \ No newline at end of file diff --git a/_images/arch-scheduler-queues.svg b/_images/arch-scheduler-queues.svg new file mode 100644 index 000000000..8732b9fc4 --- /dev/null +++ b/_images/arch-scheduler-queues.svg @@ -0,0 +1,488 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Pull + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Fetch + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Build + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Track + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Push + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Elements In + + + + + + + + + + + + + + + + + + + + Elements Out + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/arch-scheduler-run.svg b/_images/arch-scheduler-run.svg new file mode 100644 index 000000000..e031aa57e --- /dev/null +++ b/_images/arch-scheduler-run.svg @@ -0,0 +1,437 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Scheduler.run() + + + + + + + + + + + + + Wait + + + + + + Run Event Loop + + + + + + + + + + + + + Job Done + + + + + + + + ReadyElementsRemain + + + + + + + + + + + + + QueueJobs + + + + + + + + + + + + + + + + + + + + Yes + + + + + + + + + + + + + + + + + + + + Return + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + No + + + + + + + \ No newline at end of file diff --git a/_sources/CONTRIBUTING.rst.txt b/_sources/CONTRIBUTING.rst.txt new file mode 100644 index 000000000..3cad208cd --- /dev/null +++ b/_sources/CONTRIBUTING.rst.txt @@ -0,0 +1,263 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +Contributing +============ +Some tips and guidelines for developers hacking on BuildStream + + +.. _contributing_filing_issues: + +Filing issues +------------- +If you are experiencing an issue with BuildStream, or would like to submit a patch +to fix an issue, then you should first search the list of `open issues `_ +to see if the issue is already filed, and `open an issue `_ +if no issue already exists. + + +.. _contributing_fixing_bugs: + +Fixing bugs +----------- +Before fixing a bug, it is preferred that an :ref:`issue be filed ` +first in order to better document the defect, however this need not be followed to the +letter for minor fixes. + +Patches which fix bugs should always come with a regression test. + + +.. _contributing_adding_features: + +Adding new features +------------------- +Feature additions should be proposed on the `mailing list +`_ +before being considered for inclusion. To save time and avoid any frustration, +we strongly recommend proposing your new feature in advance of commencing work. + +Once consensus has been reached on the mailing list, then the proposing +party should :ref:`file an issue ` to track the +work. + +New features must be well documented and tested in our test suite. + +It is expected that the individual submitting the work take ownership +of their feature within BuildStream for a reasonable timeframe of at least +one release cycle after their work has landed on the master branch. This is +to say that the submitter is expected to address and fix any side effects, +bugs or regressions which may have fell through the cracks in the review +process, giving us a reasonable timeframe for identifying these. + + +.. _contributing_submitting_patches: + +Submitting patches +------------------ +Submitting patches works in the regular GitHub workflow of submitting +pull requests. + + +Branch names +~~~~~~~~~~~~ +If you are an apache member with access to the main repository, and are +submitting a pull request for a branch within the main repository, then +please be careful to use an identifiable branch name. + +Branch names for pull requests should be prefixed with the submitter's +name or nickname, followed by a forward slash, and then a descriptive +name. e.g.:: + + username/fix-that-bug + +This allows us to more easily identify which branch does what and +belongs to whom, especially so that we can effectively cleanup stale +branches in the upstream repository over time. + + +Pull requests +~~~~~~~~~~~~~ +Once you have created a local branch, you can push it to the upstream +BuildStream repository using the command line:: + + git push origin username/fix-that-bug + +GitHub will respond to this with a message and a link to allow you to create +a new merge request. You can also `create a pull request using the GitHub UI +`_. + +You may open pull requests for the branches you create before you are ready +to have them reviewed and considered for inclusion if you like. Until your merge +request is ready for review, the pull request title must be prefixed with the +``WIP:`` identifier. + +Consider marking a pull request as WIP again if you are taking a while to +address a review point. This signals that the next action is on you, and it +won't appear in a reviewer's search for non-WIP merge requests to review. + + + +Organized commits +~~~~~~~~~~~~~~~~~ +Submitted branches must not contain a history of the work done in the +feature branch. For example, if you had to change your approach, or +have a later commit which fixes something in a previous commit on your +branch, we do not want to include the history of how you came up with +your patch in the upstream master branch. + +Please use git's interactive rebase feature in order to compose a clean +patch series suitable for submission upstream. + +Every commit in series should pass the test suite, this is very important +for tracking down regressions and performing git bisections in the future. + +We prefer that documentation changes be submitted in separate commits from +the code changes which they document, and newly added test cases are also +preferred in separate commits. + +If a commit in your branch modifies behavior such that a test must also +be changed to match the new behavior, then the tests should be updated +with the same commit, so that every commit passes its own tests. + +These principles apply whenever a branch is non-WIP. So for example, don't push +'fixup!' commits when addressing review comments, instead amend the commits +directly before pushing. + + +Commit messages +~~~~~~~~~~~~~~~ +Commit messages must be formatted with a brief summary line, followed by +an empty line and then a free form detailed description of the change. + +The summary line must start with what changed, followed by a colon and +a very brief description of the change. + +If the commit fixes an issue, or is related to an issue; then the issue +number must be referenced in the commit message. + +**Example**:: + + element.py: Added the frobnicator so that foos are properly frobbed. + + The new frobnicator frobnicates foos all the way throughout + the element. Elements that are not properly frobnicated raise + an error to inform the user of invalid frobnication rules. + + Fixes #123 + +Note that the 'why' of a change is as important as the 'what'. + +When reviewing this, folks can suggest better alternatives when they know the +'why'. Perhaps there are other ways to avoid an error when things are not +frobnicated. + +When folks modify this code, there may be uncertainty around whether the foos +should always be frobnicated. The comments, the commit message, and issue #123 +should shed some light on that. + +In the case that you have a commit which necessarily modifies multiple +components, then the summary line should still mention generally what +changed (if possible), followed by a colon and a brief summary. + +In this case the free form detailed description of the change should +contain a bullet list describing what was changed in each component +separately. + +**Example**:: + + artifact cache: Fixed automatic expiry in the local cache + + o _artifactcache/artifactcache.py: Updated the API contract + of ArtifactCache.remove() so that something detailed is + explained here. + + o _artifactcache/cascache.py: Adhere to the new API contract + dictated by the abstract ArtifactCache class. + + o tests/artifactcache/expiry.py: Modified test expectations to + match the new behavior. + + This is a part of #123 + + +Committer access +---------------- +Committers in the BuildStream project are those folks to whom the right to +directly commit changes to our version controlled resources has been granted. + +While every contribution is valued regardless of its source, not every person +who contributes code to the project will earn commit access. +The `COMMITTERS`_ file lists all committers. + +.. _COMMITTERS: https://github.com/apache/buildstream/blob/master/COMMITTERS.rst + + +How commit access is granted +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +After someone has successfully contributed a few non-trivial patches, some full +committer, usually whoever has reviewed and applied the most patches from that +contributor, proposes them for commit access. This proposal is sent only to the +other full committers - the ensuing discussion is private, so that everyone can +feel comfortable speaking their minds. Assuming there are no objections, the +contributor is granted commit access. The decision is made by consensus; there +are no formal rules governing the procedure, though generally if someone strongly +objects the access is not offered, or is offered on a provisional basis. + +This of course relies on contributors being responsive and showing willingness +to address any problems that may arise after landing patches. However, the primary +criterion for commit access is good judgement. + +You do not have to be a technical wizard or demonstrate deep knowledge of the +entire codebase to become a committer. You just need to know what you don't +know. Non-code contributions are just as valuable in the path to commit access. +If your patches adhere to the guidelines in this file, adhere to all the usual +unquantifiable rules of coding (code should be readable, robust, maintainable, etc.), +and respect the Hippocratic Principle of "first, do no harm", then you will probably +get commit access pretty quickly. The size, complexity, and quantity of your patches +do not matter as much as the degree of care you show in avoiding bugs and minimizing +unnecessary impact on the rest of the code. Many full committers are people who have +not made major code contributions, but rather lots of small, clean fixes, each of +which was an unambiguous improvement to the code. (Of course, this does not mean the +project needs a bunch of very trivial patches whose only purpose is to gain commit +access; knowing what's worth a patch post and what's not is part of showing good +judgement.) + + +Windows CI +---------- +The infrastructure for running the CI against Windows is different from the usual +runners, due to a combination of licensing technicalities and differing +containerisation support. + +The scripts used to generate a CI runner can be found at +`https://gitlab.com/BuildStream/windows-startup-script`. +The `wsl` branch can be used to generate a runner for WSL, and the `win32` branch +can be used to generate a native-windows runner. + + +Further information +------------------- + +.. toctree:: + :maxdepth: 1 + + hacking/coding_guidelines.rst + hacking/using_the_testsuite.rst + hacking/writing_documentation.rst + hacking/writing_plugins.rst + hacking/measuring_performance.rst + hacking/making_releases.rst + hacking/grpc_protocols.rst + hacking/managing_data_files.rst + hacking/updating_python_deps.rst + hacking/ui.rst diff --git a/_sources/additional_docker.rst.txt b/_sources/additional_docker.rst.txt new file mode 100644 index 000000000..3122ee64c --- /dev/null +++ b/_sources/additional_docker.rst.txt @@ -0,0 +1,64 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +.. _bst_and_docker: + + +BuildStream and Docker +====================== +BuildStream integrates with Docker in multiple ways. Here are some ways in +which these integrations work. + + +Run BuildStream inside Docker +----------------------------- +Refer to the `BuildStream inside Docker `_ +documentation for instructions on how to run BuildStream as a Docker container. + + +Generate Docker images +---------------------- +The `bst-docker-import script `_ +can be used to generate a Docker image from built artifacts. + +You can download it and make it executable like this: + +.. code:: bash + + mkdir -p ~/.local/bin + curl --get https://raw.githubusercontent.com/apache/buildstream/master/contrib/bst-docker-import > ~/.local/bin/bst-docker-import + chmod +x ~/.local/bin/bst-docker-import + +Check if ``~/.local/bin`` appears in your PATH environment variable -- if it +doesn't, you should +`edit your ~/.profile so that it does `_. + +Once the script is available in your PATH and assuming you have Docker +installed, you can start using the ``bst-docker-import`` script. Here is a +minimal example to generate an image called ``bst-hello`` from an element +called ``hello.bst`` assuming it is already built: + +.. code:: bash + + bst-docker-import -t bst-hello hello.bst + +This script can also be used if you are running BuildStream inside Docker. In +this case, you will need to supply the command that you are using to run +BuildStream using the ``-c`` option. If you are using the +`bst-here wrapper script `_, +you can achieve the same results as the above example like this: + +.. code:: bash + + bst-docker-import -c bst-here -t bst-hello hello.bst diff --git a/_sources/arch_cachekeys.rst.txt b/_sources/arch_cachekeys.rst.txt new file mode 100644 index 000000000..4989c093d --- /dev/null +++ b/_sources/arch_cachekeys.rst.txt @@ -0,0 +1,112 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +.. _cachekeys: + + +Cache keys +========== + +Cache keys for artifacts are generated from the inputs of the build process +for the purpose of reusing artifacts in a well-defined, predictable way. + +Structure +--------- +Cache keys are SHA256 hash values generated from a UTF-8 JSON document that +includes: + +* Environment (e.g., project configuration and variables). +* Element configuration (details depend on element kind, ``Element.get_unique_key()``). +* Sources (``Source.get_unique_key()``). +* Dependencies (depending on cache key type, see below). +* Public data. + +Cache key types +--------------- +There are two types of cache keys in BuildStream, ``strong`` and ``weak``. + +The purpose of a ``strong`` cache key is to capture the state of as many aspects +as possible that can have an influence on the build output. The aim is that +builds will be fully reproducible as long as the cache key doesn't change, +with suitable module build systems that don't embed timestamps, for example. + +A ``strong`` cache key includes the strong cache key of each build dependency +(and their runtime dependencies) of the element as changes in build dependencies +(or their runtime dependencies) can result in build differences in reverse +dependencies. This means that whenever the strong cache key of a dependency +changes, the strong cache key of its reverse dependencies will change as well. + +A ``weak`` cache key has an almost identical structure, however, it includes +only the names of build dependencies, not their cache keys or their runtime +dependencies. A weak cache key will thus still change when the element itself +or the environment changes but it will not change when a dependency is updated. + +For elements without build dependencies the ``strong`` cache key is identical +to the ``weak`` cache key. + +Note that dependencies which are not required at build time do not affect +either kind of key. + +Strict build plan +----------------- +This is the default build plan that exclusively uses ``strong`` cache keys +for the core functionality. An element's cache key can be calculated when +the cache keys of the element's build dependencies (and their runtime +dependencies) have been calculated and either tracking is not enabled or it +has already completed for this element, i.e., the ``ref`` is available. +This means that with tracking disabled the cache keys of all elements could be +calculated right at the start of a build session. + +While BuildStream only uses ``strong`` cache keys with the strict build plan +for the actual staging and build process, it will still calculate ``weak`` +cache keys for each element. This allows BuildStream to store the artifact +in the cache with both keys, reducing rebuilds when switching between strict +and non-strict build plans. If the artifact cache already contains an +artifact with the same ``weak`` cache key, it's replaced. Thus, non-strict +builds always use the latest artifact available for a given ``weak`` cache key. + +Non-strict build plan +--------------------- +The non-strict build plan disables the time-consuming automatic rebuild of +reverse dependencies at the cost of dropping the reproducibility benefits. +It uses the ``weak`` cache keys for the core staging and build process. +I.e., if an artifact is available with the calculated ``weak`` cache key, +it will be reused for staging instead of being rebuilt. ``weak`` cache keys +can be calculated early in the build session. After tracking, similar to +when ``strong`` cache keys can be calculated with a strict build plan. + +Similar to how strict build plans also calculate ``weak`` cache keys, non-strict +build plans also calculate ``strong`` cache keys. However, this is slightly +more complex. To calculate the ``strong`` cache key of an element, BuildStream +requires the ``strong`` cache keys of the build dependencies (and their runtime +dependencies). + +The build dependencies of an element may have been updated since the artifact +was built. With the non-strict build plan the artifact will still be reused. +However, this means that we cannot use a ``strong`` cache key calculated purely +based on the element definitions. We need a cache key that matches the +environment at the time the artifact was built, not the current definitions. + +The only way to get the correct ``strong`` cache key is by retrieving it from +the metadata stored in the artifact. As artifacts may need to be pulled from a +remote artifact cache, the ``strong`` cache key is not readily available early +in the build session. However, it can always be retrieved when an element is +about to be built, as the dependencies are guaranteed to be in the local +artifact cache at that point. + +``Element._get_cache_key_from_artifact()`` extracts the ``strong`` cache key +from an artifact in the local cache. ``Element._get_cache_key_for_build()`` +calculates the ``strong`` cache key that is used for a particular build job. +This is used for the embedded metadata and also as key to store the artifact in +the cache. diff --git a/_sources/arch_caches.rst.txt b/_sources/arch_caches.rst.txt new file mode 100644 index 000000000..6a3e7e48a --- /dev/null +++ b/_sources/arch_caches.rst.txt @@ -0,0 +1,75 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +.. _caches: + + +Caches +====== + +BuildStream uses local caches to avoid repeating work, and can have remote +caches configured to allow the results of work to be shared between multiple +users. There are caches for both elements and sources that map keys to relevant +metadata and point to data in CAS. + +Content Addressable Storage (CAS) +--------------------------------- + +The majority of data is stored in Content Addressable Storage or CAS, which +indexes stored files by the SHA256 hash of their contents. This allows for a +flat file structure as well as any repeated data to be shared across a CAS. In +order to store directory structures BuildStream's CAS uses `protocol buffers`_ +for storing directory and file information as defined in Googles `REAPI`_. + +The data itself is stored in CAS which is defined by the `remote execution protocol`_, +and BuildStream also uses the `remote asset protocol`_ in order to address stored +content using symbolic labesl, such as :ref:`artifact names ` for +artifacts. + + +Artifact caches +--------------- + +Artifacts store build results of an element which is then referred to by its +cache key (described in :ref:`cachekeys`). The artifacts information is then +stored in a protocol buffer, defined in ``artifact.proto``, which includes +metadata such as the digest of the files root; strong and weak keys; and log +files digests. The digests point to locations in the CAS of relavant files and +directories, allowing BuildStream to query remote CAS servers for this +information. + +Source caches +------------- + +Sources are cached by running the :mod:`Source.stage +` method and capturing the directory output of +this into the CAS, which then use the sources key to refer to this. The source +key will be calculated with the plugins defined :mod:`Plugin.get_unique_key +` and, depending on whether the source +requires previous sources to be staged (e.g. the patch plugin), the unique key +of all sources listed before it in an element. Source caches are simpler than +artifacts, as they just need to map a source key to a directory digest, with no +additional metadata. + +.. note:: + + Not all plugins use the same result as the staged output for workspaces. As a + result when initialising a workspace, BuildStream may require fetching the + original source if it only has the source in the source cache. + +.. _protocol buffers: https://developers.google.com/protocol-buffers/docs/overview +.. _grpc: https://grpc.io +.. _REAPI: https://github.com/bazelbuild/remote-apis +.. _remote execution protocol: https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto +.. _remote asset protocol: https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/asset/v1/remote_asset.proto diff --git a/_sources/arch_data_model.rst.txt b/_sources/arch_data_model.rst.txt new file mode 100644 index 000000000..2d90ca338 --- /dev/null +++ b/_sources/arch_data_model.rst.txt @@ -0,0 +1,171 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Data model +========== +This section details the data model on which the BuildStream core operates. This +includes an overview of the project data model which is BuildStream's main input, +the user preferences, and local state. + + +Project +------- +The ``Project`` object is the main component of a given BuildStream *project*, and +is responsible for loading and validating the :ref:`project.conf `, and +providing this loaded *project data* in a convenient way to the BuildStream core. + +Conceptually, the *project* is a container for the :mod:`Elements `, +which are declared within a user's project, and as such acts as a factory for instantiating +elements at load time. + + +Element +------- +:mod:`Elements ` are the main processing unit in a pipeline. These +are the loaded representation of the ``.bst`` files loaded from the :ref:`project's element path +`. + +The *Element* is an abstract base class which cannot do anything on its own, its +concrete class is defined by *plugins* which are either included in the BuildStream +:ref:`core set of plugins ` or loaded from external sources :ref:`defined by the project +`. + +The responsibilities of an element include: + +* Loading the element's configuration from the core provided dictionary. +* Providing a unique key for any element specific configuration which might + affect the output produced by the element. +* Configuring the sandbox. +* Staging the data into the sandbox, which might include Sources and + the outputs of previous elements. +* Assembling the output *artifact*. + + +Element data structure +~~~~~~~~~~~~~~~~~~~~~~ +The properties of an element are a composition of what the BuildStream core understands, +the configurations exposed by the Element plugin, and free form data which allows +annotations and configurations which can be read back by reverse dependencies during +processing, as illustrated here: + +.. image:: images/arch-datamodel-element.svg + :align: center + + +Element composition +~~~~~~~~~~~~~~~~~~~ +The element is composed of configurations which are sourced from various entry +points using the low level YAML utilities. + +This composition takes place after :ref:`includes ` and +:ref:`conditional ` directives are processed, while +:ref:`list composition ` directives are processed +as a result of this composition. + +Here is a diagram showing which sources take precedence in the composition process +which results in the final element configuration being resolved: + +.. image:: images/arch-datamodel-element-composition.svg + :align: center + +Note that not all *BuildStream Core Data* is understood by the *Element*, but a great +deal of configurations understood by the *Element* is also understood by the core and +has default configurations built into BuildStream and configurable with the project +configuration. These include values such as *variables*, *environment*, *sandbox*, etc. + +As shown above, composition is performed in two stages for each element. First +we compose everything below the line, this happens just once per 'kind' of +element - the result is re-used. Secondly, we compose the element declaration +on top. + + +Source +------ +:mod:`Sources ` are the abstract objects which are responsible +for obtaining remote source code or data to import into the build environment, and +ensuring that it is done in a bit-for-bit reproducible way without any contamination +of the host or build environment. + +This is to say that: + +* User configuration on the host, or filesystem outside of BuildStream designated + directories, must never be modified as a side-effect of running BuildStream. + +* When the Source uses host tools, host side configurations must never result in + deviations of what is staged to a build directory. The Source must behave exactly + the same way regardless of host side configurations. + +The responsibilities of a source include: + +* Loading the source's configuration from the core provided dictionary. +* Providing a unique key for any source specific configuration which might + affect the staged source. +* Implement discovery of new versions of the source upstream (referred to as *"tracking"*). +* Staging the unpacked source to a given directory. +* Preparing workspaces. + + +Source data structure +~~~~~~~~~~~~~~~~~~~~~ +Similar to the *Element*, the properties of a source are a composition of what +the BuildStream core understands and the configurations exposed by the Source +plugin: + +.. image:: images/arch-datamodel-source.svg + :align: center + +.. note:: + + In .bst files, the BuildStream core configurations and Source specific configurations + share the same dictionary. + + Strictly speaking this is limiting, but provides a measure of convenience as .bst + files are a bit less wordy to express. + + +Source composition +~~~~~~~~~~~~~~~~~~ +Source composition is much simpler than Element composition, because defaults +cannot be specified at the project level, excepting for Source type specific +value overrides. + +.. image:: images/arch-datamodel-source-composition.svg + :align: center + + +Context +------- +The Context object is a very centric part of the BuildStream data model, and is +not a part of the Project data described above but rather is where we load and +store all of the user preferences. + +User preferences are sourced from various locations, but usually have a default, +an option in the user configuration file, and an option to override it on the +command line. + +.. image:: images/arch-datamodel-context.svg + :align: center + +Asides from being a focal point for loading and storing all user configuration, +the Context object also plays a central role in the logging framework. + + +Workspaces +---------- +The Workspaces object is yet another kind of state. Unlike the Context and +the Project data model, the Workspaces object loads, saves and stores in +memory the local state regarding a user's active and open workspaces. + +These are stored in the local state ``.bst/`` subdirectory of users projects. diff --git a/_sources/arch_dependency_model.rst.txt b/_sources/arch_dependency_model.rst.txt new file mode 100644 index 000000000..294491054 --- /dev/null +++ b/_sources/arch_dependency_model.rst.txt @@ -0,0 +1,85 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Dependency model +================ +Elements in the data model are related by their *dependencies*. In BuildStream, there +are two types of relationship that an Element may have with a *dependency*, +:ref:`build and runtime dependencies `. More often than not, +an element will require its dependency both to *build* and also at *runtime*. + +Consider a simple build scenario where you want to build an application, which +requires a service be present in order to function properly at *runtime*, a +compiler that need only be present at *build time*, and a runtime environment +or base system which must be required all the time: + +.. image:: images/arch-dependency-model.svg + +Note that in BuildStream we are only concerned with element level granularity +in our dependency model, and there is no way to depend on only a part of an element's +output *artifact*. Instead we can employ :mod:`compose ` and +:mod:`filter ` elements in conjunction with :ref:`split rules ` +to achieve sub artifact granularity at build and deploy time. + +When developing BuildStream, it is important to understand the distinction +between dependency types and element :class:`Scope `, +which acts as a selector of which elements to consider in the dependency +graph of a given element when performing recursive activities. + + +Scope +~~~~~ + +* **Scope.ALL** + + In the :func:`Scope.ALL ` scope, all elements + are considered. + + This is used in some cases to forcefully fetch, pull or build all dependencies + of a given element, even when not all of them are needed. + + This scope simply includes all of the dependencies, including the element itself. + +* **Scope.RUN** + + In the :func:`Scope.RUN ` scope, only elements + which are required to run are considered, including the element itself. Note + that these are transitive - the service also requires the base runtime. + + This is used when for example, launching a ``bst shell`` environment + for the purpose of running, or in any case we need to consider which + elements are required to run. + + .. image:: images/arch-dependency-model-runtime.svg + :align: center + +* **Scope.BUILD** + + In the :func:`Scope.BUILD ` scope, only + elements which are required to build are considered, *excluding* the + element we intend to build. + + .. image:: images/arch-dependency-model-build.svg + :align: center + + Note that build type dependencies are not transitive, which is why the + *Bootstrap* element is not selected when pulling in the *Compiler* to + build the *Application*. + + Further, note that we still follow the *Compiler* dependency on the + *Base Runtime*, this is because when we depend on an element for the + purpose of *building*, we expect that element to *run* and as such + we include all of the *runtime dependencies* of *build dependencies* + when selecting the *Scope.BUILD* elements. diff --git a/_sources/arch_overview.rst.txt b/_sources/arch_overview.rst.txt new file mode 100644 index 000000000..42b676f04 --- /dev/null +++ b/_sources/arch_overview.rst.txt @@ -0,0 +1,21 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Overview of modules +=================== +Below is a basic overview of the modules, what they are for, and generally +what their stacking order is internally in BuildStream. + +.. image:: images/arch-overview.svg diff --git a/_sources/arch_program_flow.rst.txt b/_sources/arch_program_flow.rst.txt new file mode 100644 index 000000000..a0ba5abc3 --- /dev/null +++ b/_sources/arch_program_flow.rst.txt @@ -0,0 +1,24 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Overview of program flow +======================== +Here is a little chart to show the approximate highlevel program flow of +BuildStream. This is in no way a complete flow chart of BuildStream, but +should provide some highlevel insight into how the program operates in +general. + +.. image:: images/arch-program-flow.svg + :align: center diff --git a/_sources/arch_remote_execution.rst.txt b/_sources/arch_remote_execution.rst.txt new file mode 100644 index 000000000..7aef44a7b --- /dev/null +++ b/_sources/arch_remote_execution.rst.txt @@ -0,0 +1,45 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +Remote execution +================ +Remote execution allows builds to take place on different machines from the machine *bst* is run on, allowing faster builds, shared resources and different build capabilities, for example different machine architectures. + +Sandbox extension +~~~~~~~~~~~~~~~~~ +The previous section :ref:`sandboxing` describes the two forms of local sandbox: the *bubblewrap* sandbox and the less common *chroot* sandbox (which is used on non-Linux POSIX environments). Remote execution uses a third type, the *remote sandbox*, which functions similarly to the local sandbox types, but is responsible for causing the build to occur on a remote system. Remote sandboxes should produce the same result as local sandboxes. + +Artifact caches and other storage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +BuildStream can transmit the results of local builds to remote artifact caches and retrieve them later. The same mechanism is used for remote execution. The filesystem staged before building is stored in a local content-addressable store, which may be the same as the local artifact cache. The command to execute is also stored as an object in local CAS. Both the initial source filesystem and command are transmitted to remote storage specific to the remote execution service, and after the build is complete, the filesystem after build is retrieved from remote storage to the local CAS. The remote execution service uses the same communication protocol as artifact caches, and may use the same internal storage, but may not implement the Remote Asset API used by BuildStream to store full artifacts. + +.. image:: images/arch-remote-execution.svg + :align: center + +After sending the prerequisite file system and command to the remote execution cache, BuildStream uses the *Remote Execution API (REAPI)* [1]_ to signal to the build server that it should perform a build. +How the build service does this is not BuildStream's concern, but typically there will be a worker that will be assigned the work, retrieve the inputs of the build from CAS, carry out the build, and store the results in CAS. The remote execution service is entitled to return a cached result, if the filesystem and command have already been executed. + + +BuildStream will continue to poll the remote execution server until the build is completed or lost. If it's completed (successfully or otherwise) the resulting objects (typically the finished file system and logs of stdout and stderr) will be pulled to the local cache. BuildStream will retry jobs that are lost by the remote build server or which complete with certain error types. + +*sandboxremote.py* contains all the communication with the remote execution API. + +After a successful build, BuildStream will push the completed artifact to the remote artifact servers as if it had built it locally. + +Use of sandboxes outside builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Because staging requires *integration-commands* to be run, a sandbox is necessary for the *bst artifact checkout* command to run. + +A sandbox is also required for the *bst shell* command. Because the REAPI does not provide any mechanism for interactive use, *bst shell* will always use a local sandbox. + +.. [1] See https://github.com/bazelbuild/remote-apis. diff --git a/_sources/arch_sandboxing.rst.txt b/_sources/arch_sandboxing.rst.txt new file mode 100644 index 000000000..a5daf3d19 --- /dev/null +++ b/_sources/arch_sandboxing.rst.txt @@ -0,0 +1,204 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +.. _sandboxing: + + +Sandboxing +========== + +Introduction +------------ + +BuildStream assembles each element in a *sandbox*. The sandbox is a container +environment which serves two purposes: giving BuildStream control over +all build aspects in order to ensure reproducibility of build results, +and providing safety guarantees for the host system that BuildStream is +running on. + +The exact implementation of the sandbox varies depending on which platform you +are running BuildStream. See below for backend-specific details. + +There are several factors that affect the build output and must therefore be +under BuildStream's control: + +* Filesystem contents and metadata +* The user and permissions model +* Network access +* Device access + +Each of these is detailed below. + +For safety reasons, BuildStream also controls the following things: + +* Access to files outside of the sandbox directory +* Access to certain kernel-specific syscalls + +Creating a sandbox can require special priviliges. This is a safety concern too +because bugs in the `bst` program can cause damage to a host if the program is +running with extra privileges. The exact priviliges that are required depend on +your platform and backend. + +Element plugins can run arbitary commands within the sandbox using the +:mod:`sandbox API `. + +What elements can and can't do in the sandbox +--------------------------------------------- + +This section specifies how BuildStream sandboxes are intended to work. A +specific sandbox provider may not necessarily be able to achieve all of the +requirements listed below so be sure to read the "platform notes" section as +well. + +Filesystem access +~~~~~~~~~~~~~~~~~ + +The filesystem inside sandboxes should be read-only during element assembly, +except for certain directories which element plugins can mark as being +read/write. Most elements plugins derive from :mod:`BuildElement +`, which marks ``%{build-root}`` and +``%{install-root}`` as read/write. + +When running integration commands or `bst shell`, the sandbox should have a +fully read-write filesystem. The changes made here do not need to persist +beyond the lifetime of that sandbox, and **must not** affect the contents of +artifacts stored in the cache. + +Certain top level directories should be treated specially in all sandboxes: + +* The ``/dev`` directory should contain device nodes, which are described in + a separate section. + +* The ``/proc`` directory should have a UNIX 'procfs' style filesystem mounted. + It should not expose any information about processes running outside of the + sandbox. + +* The ``/tmp`` directory should be writable. + +Filesystem metadata +~~~~~~~~~~~~~~~~~~~ + +The writable areas inside a BuildStream sandbox are limited in what metadata +can be written and stored. + +* All files must be owned by UID 0 and GID 0 +* No files may have the setuid or setgid bits set +* Extended file attributes (xattrs) cannot be written to or read. +* Hardlinks to other files can be created, but the information about which + files are hardlinked to each other will not be stored in the artifact + that is created from the sandbox. + +These restrictions are due to technical limitations. In future we hope to +support a wider range of filesystem metadata operations. See `issue #38 +`_ for more details. + +User and permissions model +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All commands inside the sandbox run with user ID 0 and group ID 0. It should +not be possible to become any other user ID. + +Network access +~~~~~~~~~~~~~~ + +Builds should not be able to access the network at all from the sandbox. All +remote resources needed to build an element must be specified in the element's +``sources`` list so that BuildStream is able to see when they have changed. + +A sandbox opened by `bst shell` should allow network access. + +Device access +~~~~~~~~~~~~~ + +Builds should not be able to access any hardware devices at all. + +A few standard UNIX device files are needed, the whitelist is: + +* ``/dev/full`` +* ``/dev/null`` +* ``/dev/urandom`` +* ``/dev/random`` +* ``/dev/zero`` + +It may seem odd that we have sources of randomness in the sandbox, but a lot of +tools do expect them to exist. We take the view that it's up to integrators to +ensure that elements do not deliberately include randomness in their output. + +A sandbox opened by `bst shell` can make any devices available. There needs to +be a console device so that it can be used interactively. + +Platform notes +-------------- + +BuildStream delegates sandboxing for local builds to the ``buildbox-run`` +command. ``buildbox-run`` provides a platform-independent interface to execute +commands in a sandbox based on parts of the Remote Execution API. + +Linux +~~~~~ + +The recommended ``buildbox-run`` implementation for Linux is +``buildbox-run-bubblewrap``, in combination with ``buildbox-fuse``. + +These implementations use the following isolation and sandboxing primitives: + +* bind mounts +* FUSE +* Mount namespaces +* Network namespaces +* PID (process ID) namespaces +* User namespaces (if available) +* seccomp + +We access all of these features through a sandboxing tool named `Bubblewrap +`_. + +User namespaces are not enabled by default in all Linux distributions. +BuildStream still runs on such systems but can't build projects that set +``build-uid`` or ``build-gid`` in the ``sandbox`` configuration. + +The Linux platform can operate as a standard user, if unprivileged user namespace +support is available. If user namespace support is not available you have the +option of installing bubblewrap as a setuid binary to avoid needing to run the +entire ``bst`` process as the ``root`` user. + +FUSE is used to provide access to directories and files stored in CAS without +having to copy or hardlink the complete input tree into a regular filesystem +directory structure for each build job. + +Some of the operations on filesystem metadata listed above are not prohibited +by the sandbox, but will instead be silently dropped when an artifact is +created. For more details see `issue #38 +`_. + +Some details of the host machine are currently leaked by this platform backend. +For more details, see `issue #262 +`_. + +Other POSIX systems +~~~~~~~~~~~~~~~~~~~ + +On other POSIX systems ``buildbox-run-userchroot`` may be used for sandboxing. +`userchroot `_ allows regular +users to invoke processes in a chroot environment. + +``buildbox-run-userchroot`` stages the input tree for each build job using +hardlinks to avoid more expensive file copies. To avoid cache corruption it is +vital that hardlinked files cannot be overwritten. Due to this it's required +to run ``buildbox-casd`` as a separate user, which owns the files in the local +cache. + +Network access is not blocked in the chroot. However since there is unlikely +to be a correct `/etc/resolv.conf` file, any network access that depends on +name resolution will most likely fail anyway. diff --git a/_sources/arch_scheduler.rst.txt b/_sources/arch_scheduler.rst.txt new file mode 100644 index 000000000..572bf599a --- /dev/null +++ b/_sources/arch_scheduler.rst.txt @@ -0,0 +1,145 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Scheduler +========= +The *Scheduler* is what is responsible for running the main event loop and +dispatching *Jobs* to complete tasks on behalf of *Queues*. + + +Jobs +~~~~ +The basic functionality of multiprocessing tasks is implemented by the base Job +class, which is derived in a few ways but for now we'll only talk about the +ElementJob type since that is the most centric. + +The Job class has the following responsibilities: + +* Starting the given job as a subprocess. + +* Offering an abstract method for subclasses to handle the outcome of + a job when it completes. + +* Forcefully terminating its subprocess. + +* Suspending and resuming its subprocess. + +* Declaring the types of resources it will require, and which resources + it will require exclusively. + + +Below is a rough outline of the interactions between the main process +and job specific child process: + +.. image:: images/arch-scheduler-job.svg + :align: center + + +Resources +~~~~~~~~~ +To understand how we manage load balancing in the scheduler it is important +to understand *resources*. + +For the scheduler, *resources* are domains which a Job can request which represent +physical resources, such as the CPU or some network bandwidth, or the local +artifact cache. + +This is used by the Scheduler when consuming Jobs from Queues and deciding +how many jobs can be run at a given time. + + +Queues +~~~~~~ +The various Queue implementations in the Scheduler can be combined such that +parallelism is maximized. For example one can *Track* and *Build* inside the +same session, in this way one does not need to wait for a tracking session to +complete in order to start building. + +The input elements to the scheduler are expected to be sorted in depth first +order whenever the order is important, again allowing maximum parallelism +at build time. + +.. image:: images/arch-scheduler-queues.svg + :align: center + +The Queue implementations are: + +* **Track** + + The tracking queue must always come first if it is used in the given session. + This is because the Source *"ref"*, and consequently the cache key of any elements + which have been requested for tracking, cannot be known until tracking is complete. + +* **Pull** + + The pull queue tries to obtain a built artifact from a remote artifact server, + it should be placed in advance of the fetch queue if used, since we can safely + avoid fetching if fetching is not imperative, and we already have a cached + artifact. + +* **Fetch** + + The fetch queue attempts to download source code to build the given element, + this activity is sometimes skipped if the artifact is already present, or + if the exact source code is already present. + +* **Build** + + The build queue attempts to build the element if its artifact is not locally + present. + +* **Push** + + The push queue attempts to push the resulting artifact to a remote artifact + server. + + +Queue internals +~~~~~~~~~~~~~~~ +Internally, the queue has an input queue and an output queue. + +.. image:: images/arch-scheduler-queue-ports.svg + :align: center + +When elements are on the input queue they get queried for their *QueueStatus* +in order to determine which elements should be processed or moved from the input +queue to the output queue. When elements are on the output queue, they are ready +to be consumed by the scheduler and moved on to the next queue; however each +queue holds on to the result status of every element which passed through for later +reference and reports to the user. + + +Scheduler +~~~~~~~~~ +The scheduler itself has the main responsibility of popping off jobs from +the existing queues it was given, and running the jobs as long as elements +remain to be processed. + +A huge simplification of this can be visualized as follows: + +.. image:: images/arch-scheduler-run.svg + :align: center + +This is implemented by iterating over the Queues given to the scheduler +and processing any *"Ready"* elements so long as there are sufficient resource +tokens available for the ready elements to run, and by moving the *"Done"* +elements onto the subsequent queues in the list of queues. + +.. note:: + + When looking for *"Ready"* elements in the queues, we iterate over the + queue list in *reverse order*. This is important to allow elements to + get as far as they can in the queue list as fast as possible, and helps + to prevent resource starvation. diff --git a/_sources/buildstream.buildelement.rst.txt b/_sources/buildstream.buildelement.rst.txt new file mode 100644 index 000000000..2fe3a2051 --- /dev/null +++ b/_sources/buildstream.buildelement.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.buildelement + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.downloadablefilesource.rst.txt b/_sources/buildstream.downloadablefilesource.rst.txt new file mode 100644 index 000000000..69efff3e9 --- /dev/null +++ b/_sources/buildstream.downloadablefilesource.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.downloadablefilesource + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.element.rst.txt b/_sources/buildstream.element.rst.txt new file mode 100644 index 000000000..9630b1537 --- /dev/null +++ b/_sources/buildstream.element.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.element + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.exceptions.rst.txt b/_sources/buildstream.exceptions.rst.txt new file mode 100644 index 000000000..486d82192 --- /dev/null +++ b/_sources/buildstream.exceptions.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.exceptions + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.node.rst.txt b/_sources/buildstream.node.rst.txt new file mode 100644 index 000000000..fd43fb9c0 --- /dev/null +++ b/_sources/buildstream.node.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.node + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.plugin.rst.txt b/_sources/buildstream.plugin.rst.txt new file mode 100644 index 000000000..9bbb8f4b9 --- /dev/null +++ b/_sources/buildstream.plugin.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.plugin + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.rst.txt b/_sources/buildstream.rst.txt new file mode 100644 index 000000000..7d7208a07 --- /dev/null +++ b/_sources/buildstream.rst.txt @@ -0,0 +1,34 @@ +buildstream package +=================== + +.. automodule:: buildstream + :members: + :undoc-members: + :show-inheritance: + +Subpackages +----------- + +.. toctree:: + :maxdepth: 4 + + buildstream.sandbox + buildstream.storage + +Submodules +---------- + +.. toctree:: + :maxdepth: 4 + + buildstream.buildelement + buildstream.downloadablefilesource + buildstream.element + buildstream.exceptions + buildstream.node + buildstream.plugin + buildstream.scriptelement + buildstream.source + buildstream.sourcemirror + buildstream.types + buildstream.utils diff --git a/_sources/buildstream.sandbox.rst.txt b/_sources/buildstream.sandbox.rst.txt new file mode 100644 index 000000000..7b39c6f96 --- /dev/null +++ b/_sources/buildstream.sandbox.rst.txt @@ -0,0 +1,15 @@ +buildstream.sandbox package +=========================== + +.. automodule:: buildstream.sandbox + :members: + :undoc-members: + :show-inheritance: + +Submodules +---------- + +.. toctree:: + :maxdepth: 4 + + buildstream.sandbox.sandbox diff --git a/_sources/buildstream.sandbox.sandbox.rst.txt b/_sources/buildstream.sandbox.sandbox.rst.txt new file mode 100644 index 000000000..d964cce64 --- /dev/null +++ b/_sources/buildstream.sandbox.sandbox.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.sandbox.sandbox + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.scriptelement.rst.txt b/_sources/buildstream.scriptelement.rst.txt new file mode 100644 index 000000000..f21fe26e4 --- /dev/null +++ b/_sources/buildstream.scriptelement.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.scriptelement + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.source.rst.txt b/_sources/buildstream.source.rst.txt new file mode 100644 index 000000000..b620c92a1 --- /dev/null +++ b/_sources/buildstream.source.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.source + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.sourcemirror.rst.txt b/_sources/buildstream.sourcemirror.rst.txt new file mode 100644 index 000000000..62f438c08 --- /dev/null +++ b/_sources/buildstream.sourcemirror.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.sourcemirror + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.storage.directory.rst.txt b/_sources/buildstream.storage.directory.rst.txt new file mode 100644 index 000000000..c2c6c603c --- /dev/null +++ b/_sources/buildstream.storage.directory.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.storage.directory + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.storage.rst.txt b/_sources/buildstream.storage.rst.txt new file mode 100644 index 000000000..56135d892 --- /dev/null +++ b/_sources/buildstream.storage.rst.txt @@ -0,0 +1,15 @@ +buildstream.storage package +=========================== + +.. automodule:: buildstream.storage + :members: + :undoc-members: + :show-inheritance: + +Submodules +---------- + +.. toctree:: + :maxdepth: 4 + + buildstream.storage.directory diff --git a/_sources/buildstream.types.rst.txt b/_sources/buildstream.types.rst.txt new file mode 100644 index 000000000..9b762175c --- /dev/null +++ b/_sources/buildstream.types.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.types + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/buildstream.utils.rst.txt b/_sources/buildstream.utils.rst.txt new file mode 100644 index 000000000..4f9a774ce --- /dev/null +++ b/_sources/buildstream.utils.rst.txt @@ -0,0 +1,4 @@ +.. automodule:: buildstream.utils + :members: + :undoc-members: + :show-inheritance: diff --git a/_sources/core_additional.rst.txt b/_sources/core_additional.rst.txt new file mode 100644 index 000000000..b49675400 --- /dev/null +++ b/_sources/core_additional.rst.txt @@ -0,0 +1,22 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Additional writings +=================== + +.. toctree:: + :maxdepth: 2 + + additional_docker diff --git a/_sources/core_format.rst.txt b/_sources/core_format.rst.txt new file mode 100644 index 000000000..e8089a95f --- /dev/null +++ b/_sources/core_format.rst.txt @@ -0,0 +1,30 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Project format +============== +This section details how to use the BuildStream YAML format to +create your own project or modify existing projects. + + +.. toctree:: + :maxdepth: 2 + :caption: Project format + + format_intro + format_project + format_declaring + format_public + format_project_refs diff --git a/_sources/core_framework.rst.txt b/_sources/core_framework.rst.txt new file mode 100644 index 000000000..e9777266d --- /dev/null +++ b/_sources/core_framework.rst.txt @@ -0,0 +1,40 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _core_framework: + +Plugin API reference +==================== +The core public APIs are of interest to anyone who wishes to +implement custom :mod:`Element ` or +:mod:`Source ` plugins, and can also be +useful for working on BuildStream itself. + +.. toctree:: + :maxdepth: 1 + + buildstream.types + buildstream.node + buildstream.plugin + buildstream.source + buildstream.sourcemirror + buildstream.downloadablefilesource + buildstream.element + buildstream.buildelement + buildstream.scriptelement + buildstream.sandbox.sandbox + buildstream.storage.directory + buildstream.exceptions + buildstream.utils diff --git a/_sources/core_plugins.rst.txt b/_sources/core_plugins.rst.txt new file mode 100644 index 000000000..4d52416f0 --- /dev/null +++ b/_sources/core_plugins.rst.txt @@ -0,0 +1,73 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +.. _plugins: + +Plugin specific documentation +============================= +Plugins provide their own individual plugin specific YAML configurations, +The element ``.bst`` files can specify plugin specific configuration in +the :ref:`config section `, while sources declared on a +given element specify their plugin specific configuration directly +:ref:`in their source declarations `. + + +.. _plugins_elements: + +Elements +-------- +.. toctree:: + :maxdepth: 1 + + elements/stack + elements/import + elements/compose + elements/script + elements/link + elements/junction + elements/filter + elements/manual + + +.. _plugins_sources: + +Sources +------- + +All source plugins can be staged into an arbitrary directory within the build +sandbox with the ``directory`` option. +See :ref:`Source class built-in functionality ` for more +information. + +.. toctree:: + :maxdepth: 1 + + sources/local + sources/remote + sources/tar + + +.. _plugins_external: + +External plugins +---------------- +External plugins need to be :ref:`loading through junctions `, +or alternatively installed separately in the python environment where you are +running BuildStream and loaded using the :ref:`pip method `. + +Here is a list of BuildStream plugin projects known to us at this time: + +* `buildstream-plugins `_ +* `buildstream-plugins-community (formerly known as bst-plugins-experimental) `_ +* `bst-plugins-container `_ diff --git a/_sources/developing/strict-mode.rst.txt b/_sources/developing/strict-mode.rst.txt new file mode 100644 index 000000000..ed3255ce2 --- /dev/null +++ b/_sources/developing/strict-mode.rst.txt @@ -0,0 +1,264 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _developing_strict_mode: + +Strict mode +=========== +In this section, we will cover the usage of :ref:`strict vs non-strict ` +build plans in conjunction with :ref:`workspaces `, and how this +can help to improve your edit/compile/test cycles. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/strict-mode + `_ + subdirectory. + + +Overview +-------- +When working with BuildStream to create integrations, it is typical that you have a +lot of components to build, and you frequently need to modify a component +at various levels of the stack. When developing one or more applications, you might +want to open a workspace and fix a bug in an application, or you might need to +open a workspace on a low level shared library to fix the behavior of one or +more misbehaving applications. + +By default, BuildStream will always choose to be deterministic in order to +produce the most correct build results as possible. As such, modifying a low +level library will result in rebuilding all of it's reverse dependencies, but +this can be very time consuming and inconvenient for your edit/compile/test +cycles. + +This is when enabling :ref:`non-strict build plans ` +can be helpful. + +To illustrate the facets of how this works, this example will present a project +consisting of an application which is linked both statically and dynamically +linked to a common library. + + +Project structure +----------------- +This project is mostly based on the :ref:`integration commands ` +example, as such we will ignore large parts of this project and only focus +on the elements which are of specific interest. + +To illustrate the relationship of these two applications and the library, +let's briefly take a look at the underlying Makefiles which are used in this +project, starting with the library and followed by both Makefiles used to +build the application. + + +``files/libhello/Makefile`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/strict-mode/files/libhello/Makefile + :language: Makefile + + +``files/hello/Makefile.dynamic`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/strict-mode/files/hello/Makefile.dynamic + :language: Makefile + + +``files/hello/Makefile.static`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/strict-mode/files/hello/Makefile.static + :language: Makefile + +As we can see, we have a library that is distributed both as the dynamic +library ``libhello.so`` and also as the static archive ``libhello.a``. + +Now let's take a look at the two separate elements which build the +application, first the dynamically linked version and then the static one. + + +``elements/hello-dynamic.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/strict-mode/elements/hello-dynamic.bst + :language: yaml + +Nothing very special to observe about this hello program, just a +:mod:`manual ` element quite similar to the one we've +already seen in the :ref:`running commands ` +example. + + +``elements/hello-static.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/strict-mode/elements/hello-static.bst + :language: yaml + +Almost the same as the dynamic element, except here we have declared the +dependency to the ``libhello.bst`` element differently: this time we have enabled +the ``strict`` option in the :ref:`dependency declaration `. + +The side effect of setting this option is that ``hello-static.bst`` will be +rebuilt any time that ``libhello.bst`` has changed, even when +:ref:`non-strict build plans ` have been enabled. + +.. tip:: + + Some element plugins are designed to consume the content of their + dependencies entirely, and output an artifact without any transient + runtime dependencies, an example of this is the :mod:`compose ` + element. + + In cases such as :mod:`compose `, it is not necessary to + explicitly annotate their dependencies as ``strict``. + + It is only helpful to set the ``strict`` attribute on a + :ref:`dependency declaration ` in the case that the + specific dependency relationship causes data to be consumed verbatim, + as is the case with static linking. + + +Using the project +----------------- +For the sake of brevity, let's assume that you've already built all of the +elements of this project, and that you want to make some changes to the +``libhello.bst`` element, and test how it might effect the hello program. + + +Everything is already built +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. raw:: html + :file: ../sessions/strict-mode-show-initial.html + + +Open a workspace and modify libhello.c +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Now let's open up a workspace on the hello library + +.. raw:: html + :file: ../sessions/strict-mode-workspace-open.html + +And go ahead and make a modification like this: + +.. literalinclude:: ../../examples/strict-mode/update.patch + :language: diff + + +Observing ``hello-dynamic.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Let's take a look at the :ref:`bst show ` output for +the dynamically linked ``hello-dynamic.bst`` element. + +.. raw:: html + :file: ../sessions/strict-mode-show-dynamic-strict.html + +As one might expect, the ``libhello.bst`` element is ready to be built +after having been modified, and the ``hello-dynamic.bst`` element is +waiting for ``libhello.bst`` to be built before it can build. + +Now let's take a look at the same elements if we pass the ``--no-strict`` +option to ``bst``: + +.. raw:: html + :file: ../sessions/strict-mode-show-dynamic-no-strict.html + +Note that this time, the ``libhello.bst`` still needs to be built, +but the ``hello-dymamic.bst`` element is showing up as ``cached``. + +.. tip:: + + The :ref:`bst show ` output will show some cache + keys dimmed out in the case that they are not entirely deterministic. + + Here we can see that ``hello-dynamic.bst`` is dimmed out because + it will not be rebuilt against the changed ``libhello.bst`` element, + and it also has a different cache key because of this. + + +Observing ``hello-static.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Now let's observe the ``hello-static.bst`` element with strict mode +disabled: + +.. raw:: html + :file: ../sessions/strict-mode-show-static-no-strict.html + +Note that in this case the ``hello-strict.bst`` is going to be +rebuilt even in strict mode. This is because we annotated the +declaration of the ``libhello.bst`` dependency with the ``strict`` +attribute. + +We did this because ``hello-strict.bst`` consumes the input of +``libhello.bst`` verbatim, by way of statically linking to it, instead +of merely being affected by the content of ``libhello.bst`` at runtime, +as would be the case of static linking. + + +Building and running ``hello-dynamic.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Now let's build ``hello-dynamic.bst`` with strict mode disabled. + +.. raw:: html + :file: ../sessions/strict-mode-build-dynamic-no-strict.html + +Note that the :ref:`bst build ` command completed without +having to build ``hello-dynamic.bst`` at all. + +And now we can also run ``hello-dynamic.bst`` + +.. raw:: html + :file: ../sessions/strict-mode-run-dynamic-no-strict.html + +When running ``hello-dynamic.bst`` with no-strict mode, we are +actually reusing the old build of ``hello-dynamic.bst`` staged against +the new build of the modified ``libhello.bst`` element. + + +Building and running ``hello-static.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Finally, if we build ``hello-static.bst`` with strict mode disabled, +we can see that it will be rebuilt regardless of strict mode being +enabled. + +.. raw:: html + :file: ../sessions/strict-mode-build-static-no-strict.html + +This is of course because we declared its dependency on ``libhello.bst`` +as a ``strict`` dependency. + +And by the same virtue, we can see that when we run the example +it has properly relinked against the changed static archive, and +has the updated text in the greeting: + +.. raw:: html + :file: ../sessions/strict-mode-run-static-no-strict.html + + +Summary +------- +In this chapter we've explored how to use :ref:`non-strict build plans ` +in order to avoid rebuilding reverse dependencies of a lower level +element you might be working with in a :ref:`workspace `, +consequently improving your edit/compile/test experience. + +We've also explained how to ensure your project still works properly +with non-strict build plans when some elements perform static linking +(or other operations which consume data from their dependencies +verbatim), by annotating :ref:`dependency declarations ` +as ``strict``. diff --git a/_sources/developing/workspaces.rst.txt b/_sources/developing/workspaces.rst.txt new file mode 100644 index 000000000..db2c9a95e --- /dev/null +++ b/_sources/developing/workspaces.rst.txt @@ -0,0 +1,150 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _developing_workspaces: + +Workspaces +========== +In this section we will cover the use of BuildStream's workspaces feature when +devloping a BuildStream project. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/developing + `_ + subdirectory. + +We will start with the project used in the :ref:`running commands ` +tutorial. Recall the element hello.bst, which builds the bellow C file: + +.. literalinclude:: ../../examples/developing/files/src/hello.c + :language: c + +Suppose we now want to alter the functionality of the *hello* command. We can +make changes to the source code of Buildstream elements by making use of +BuildStream's workspace command. + + +Opening a workspace +------------------- +First we need to open a workspace, we can do this by running + +.. raw:: html + :file: ../sessions/developing-workspace-open.html + +This command has created the workspace_hello directory in which you can see +the source for the hello.bst element, i.e. hello.c and the corresponding +makefile. + +You can view existing workspaces using + +.. raw:: html + :file: ../sessions/developing-workspace-list.html + + +Making code changes +------------------- +Let's say we want to alter the message printed when the hello command is run. +We can open workspace_hello/hello.c and make the following change: + +.. literalinclude:: ../../examples/developing/update.patch + :language: diff + +Now, rebuild the hello.bst element. + +.. raw:: html + :file: ../sessions/developing-build-after-changes.html + +Note that if you run the command from inside the workspace, the element name is optional. + +.. raw:: html + :file: ../sessions/developing-build-after-changes-workspace.html + +Now running the hello command using bst shell: + +.. raw:: html + :file: ../sessions/developing-shell-after-changes.html + +This gives us the new message we changed in hello.c. + +From this point we have several options. If the source is under version control +we can commit our changes and push them to the remote repository. + + +Incremental builds +------------------ +Once you have opened up your workspace, the workspace build directory will be +reused for subsequent builds, which should improve your edit/compile/test cycle +time when working with an open workspace. + +In order to optimize incremental builds, BuildStream treats build configure steps +separately from the main build steps, and will only call the +:func:`Element.prepare() ` method on +an element plugin the first time it gets built. This avoids needlessly rebuilding +objects due to header files and such being unconditionally recreated by configuration +scripts (such as the typical ``./configure`` script which is called for ``autotools`` +elements for instance). + +A caveat of this optimization however is that changes you might make to the +configuration scripts will not be taken into account by default on the next +incremental build. A forced reconfiguration can also be required in some cases +where build scripts automatically detect sources in it's configuration phase, +so newly added sources you add might be ignored. + +In order to force the configuration step to be called again on the next build, +you can use :ref:`bst workspace reset --soft `, like so: + +In these cases, you can perform a hard reset on the workspace using +:ref:`bst workspace reset `, like so: + +.. raw:: html + :file: ../sessions/developing-soft-reset.html + +This will ensure that the next time you run the build, BuildStream will forcefully +rerun the :func:`Element.prepare() ` method +and cause the configuration step to occur again. + + +Closing your workspace +---------------------- +If we want to close the workspace and come back to our changes later, we can + +.. raw:: html + :file: ../sessions/developing-close-workspace.html + +We can then reopen the workspace later using: + +.. raw:: html + :file: ../sessions/developing-reopen-workspace.html + +The --no-checkout option tells BuildStream not to check the source out but to +instead hard-link to the workspace_hello directory. + +Alternatively, if we wish to discard the changes we can use + +.. raw:: html + :file: ../sessions/developing-reset-workspace.html + +This resets the workspace to its original state. + +To discard the workspace completely we can do: + +.. raw:: html + :file: ../sessions/developing-discard-workspace.html + +This will close the workspace and completely remove the workspace_hello +directory. + diff --git a/_sources/elements/compose.rst.txt b/_sources/elements/compose.rst.txt new file mode 100644 index 000000000..7b25830c1 --- /dev/null +++ b/_sources/elements/compose.rst.txt @@ -0,0 +1 @@ +.. automodule:: elements.compose diff --git a/_sources/elements/filter.rst.txt b/_sources/elements/filter.rst.txt new file mode 100644 index 000000000..084674a21 --- /dev/null +++ b/_sources/elements/filter.rst.txt @@ -0,0 +1 @@ +.. automodule:: elements.filter diff --git a/_sources/elements/import.rst.txt b/_sources/elements/import.rst.txt new file mode 100644 index 000000000..55c2a866b --- /dev/null +++ b/_sources/elements/import.rst.txt @@ -0,0 +1 @@ +.. automodule:: elements.import diff --git a/_sources/elements/junction.rst.txt b/_sources/elements/junction.rst.txt new file mode 100644 index 000000000..ff5fca00f --- /dev/null +++ b/_sources/elements/junction.rst.txt @@ -0,0 +1 @@ +.. automodule:: elements.junction diff --git a/_sources/elements/link.rst.txt b/_sources/elements/link.rst.txt new file mode 100644 index 000000000..e461dfa57 --- /dev/null +++ b/_sources/elements/link.rst.txt @@ -0,0 +1 @@ +.. automodule:: elements.link diff --git a/_sources/elements/manual.rst.txt b/_sources/elements/manual.rst.txt new file mode 100644 index 000000000..16b7122b6 --- /dev/null +++ b/_sources/elements/manual.rst.txt @@ -0,0 +1 @@ +.. automodule:: elements.manual diff --git a/_sources/elements/script.rst.txt b/_sources/elements/script.rst.txt new file mode 100644 index 000000000..b2952b41a --- /dev/null +++ b/_sources/elements/script.rst.txt @@ -0,0 +1 @@ +.. automodule:: elements.script diff --git a/_sources/elements/stack.rst.txt b/_sources/elements/stack.rst.txt new file mode 100644 index 000000000..1150ff1e9 --- /dev/null +++ b/_sources/elements/stack.rst.txt @@ -0,0 +1 @@ +.. automodule:: elements.stack diff --git a/_sources/examples/git-mirror.rst.txt b/_sources/examples/git-mirror.rst.txt new file mode 100644 index 000000000..bcce6ec88 --- /dev/null +++ b/_sources/examples/git-mirror.rst.txt @@ -0,0 +1,157 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Creating and using a git mirror +''''''''''''''''''''''''''''''' +This is an example of how to create a git mirror using git's +`git-http-backend `_ and +`lighttpd `_. + + +Prerequisites +============= +You will need git installed, and git-http-backend must be present. It is assumed +that the git-http-backend binary exists at `/usr/lib/git-core/git-http-backend`. + +You will need `lighttpd` installed, and at the bare minimum has the modules +`mod_alias`, `mod_cgi`, and `mod_setenv`. + +I will be using gnome-modulesets as an example, which can be cloned from +`http://gnome7.codethink.co.uk/gnome-modulesets.git`. + + +Starting a git http server +========================== + + +1. Set up a directory containing mirrors +---------------------------------------- +Choose a suitable directory to hold your mirrors, e.g. `/var/www/git`. + +Place the git repositories you want to use as mirrors in the mirror dir, e.g. +``git clone --mirror http://git.gnome.org/browse/yelp-xsl /var/www/git/yelp-xsl.git``. + + +2. Configure lighttpd +--------------------- +Write out a lighttpd.conf as follows: + +:: + + server.document-root = "/var/www/git/" + server.port = 3000 + server.modules = ( + "mod_alias", + "mod_cgi", + "mod_setenv", + ) + + alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" ) + $HTTP["url"] =~ "^/git" { + cgi.assign = ("" => "") + setenv.add-environment = ( + "GIT_PROJECT_ROOT" => "/var/www/git", + "GIT_HTTP_EXPORT_ALL" => "" + ) + } + +.. note:: + + If you have your mirrors in another directory, replace /var/www/git/ with that directory. + + +3. Start lighttpd +----------------- +lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``. + + +4. Test that you can fetch from it +---------------------------------- +We can then clone the mirrored repo using git via http with +``git clone http://127.0.0.1:3000/git/yelp-xsl``. + +.. note:: + + If you have set server.port to something other than the default, you will + need to replace the '3000' in the command-line. + + +5. Configure the project to use the mirror +------------------------------------------ +To add this local http server as a mirror, add the following to the project.conf: + +.. code:: yaml + + mirrors: + - name: local-mirror + aliases: + git_gnome_org: + - http://127.0.0.1:3000/git/ + + +6. Test that the mirror works +----------------------------- +We can make buildstream use the mirror by setting the alias to an invalid URL, e.g. + +.. code:: yaml + + aliases: + git_gnome_org: https://www.example.com/invalid/url/ + +Now, if you build an element that uses the source you placed in the mirror +(e.g. ``bst build core-deps/yelp-xsl.bst``), you will see that it uses your mirror. + + +.. _lighttpd_git_tar_conf: + +Bonus: lighttpd conf for git and tar +==================================== +For those who have also used the :ref:`tar-mirror tutorial `, +a combined lighttpd.conf is below: + +:: + + server.document-root = "/var/www/" + server.port = 3000 + server.modules = ( + "mod_alias", + "mod_cgi", + "mod_setenv", + ) + + alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" ) + $HTTP["url"] =~ "^/git" { + cgi.assign = ("" => "") + setenv.add-environment = ( + "GIT_PROJECT_ROOT" => "/var/www/git", + "GIT_HTTP_EXPORT_ALL" => "" + ) + } else $HTTP["url"] =~ "^/tar" { + dir-listing.activate = "enable" + } + + +Further reading +=============== +If this mirror isn't being used exclusively in a secure network, it is strongly +recommended you `use SSL `_. + +This is the bare minimum required to set up a git mirror. A large, public project +would prefer to set it up using the +`git protocol `_, +and a security-conscious project would be configured to use +`git over SSH `_. + +Lighttpd is documented on `its wiki `_. diff --git a/_sources/examples/tar-mirror.rst.txt b/_sources/examples/tar-mirror.rst.txt new file mode 100644 index 000000000..77146b1ad --- /dev/null +++ b/_sources/examples/tar-mirror.rst.txt @@ -0,0 +1,116 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _using_tar_mirror: + +Creating and using a tar mirror +''''''''''''''''''''''''''''''' +This is an example of how to create a tar mirror using +`lighttpd `_. + + +Prerequisites +============= +You will need `lighttpd` installed. + + +I will be using gnome-modulesets as an example, which can be cloned from +`http://gnome7.codethink.co.uk/gnome-modulesets.git`. + + +Starting a tar server +===================== + + +1. Set up a directory containing mirrors +---------------------------------------- +Choose a suitable directory to hold your mirrored tar files, e.g. `/var/www/tar`. + +Place the tar files you want to use as mirrors in your mirror dir, e.g. + +.. code:: + + mkdir -p /var/www/tar/gettext + wget -O /var/www/tar/gettext/gettext-0.19.8.1.tar.xz https://ftp.gnu.org/gnu/gettext/gettext-0.19.8.1.tar.xz + + +2. Configure lighttpd +--------------------- +Write out a lighttpd.conf as follows: + +:: + + server.document-root = "/var/www/tar/" + server.port = 3000 + + dir-listing.activate = "enable" + +.. note:: + + If you have your mirrors in another directory, replace /var/www/tar/ with that directory. + +.. note:: + + An example lighttpd.conf that works for both git and tar services is available + :ref:`here ` + + +3. Start lighttpd +----------------- +lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``. + + +4. Test that you can fetch from it +---------------------------------- +We can then download the mirrored file with ``wget 127.0.0.1:3000/tar/gettext/gettext-0.19.8.1.tar.xz``. + +.. note:: + + If you have set server.port to something other than the default, you will need + to replace the '3000' in the command-line. + + +5. Configure the project to use the mirror +------------------------------------------ +To add this local http server as a mirror, add the following to the project.conf: + +.. code:: yaml + + mirrors: + - name: local-mirror + aliases: + ftp_gnu_org: + - http://127.0.0.1:3000/tar/ + + +6. Test that the mirror works +----------------------------- +We can make buildstream use the mirror by setting the alias to an invalid URL, e.g. + +.. code:: yaml + + aliases: + ftp_gnu_org: https://www.example.com/invalid/url/ + +Now, if you build an element that uses the source you placed in the mirror +(e.g. ``bst build core-deps/gettext.bst``), you will see that it uses your mirror. + + +Further reading +=============== +If this mirror isn't being used exclusively in a secure network, it is strongly +recommended you `use SSL `_. + +Lighttpd is documented on `its wiki `_. diff --git a/_sources/format_declaring.rst.txt b/_sources/format_declaring.rst.txt new file mode 100644 index 000000000..7bad61f12 --- /dev/null +++ b/_sources/format_declaring.rst.txt @@ -0,0 +1,728 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Declaring elements +================== + + +.. _format_basics: + +Element basics +-------------- +Here is a rather complete example using the autotools element kind and git source kind: + +.. code:: yaml + + # Specify the kind of element this is + kind: autotools + + # Specify some dependencies + depends: + - element1.bst + - element2.bst + + # Specify the source which should be built + sources: + - kind: git + url: upstream:modulename.git + track: master + ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 + + # Override some variables + variables: + sysconfdir: "%{prefix}/etc" + + # Tweak the sandbox shell environment + environment: + LD_LIBRARY_PATH: /some/custom/path + + # Specify the configuration of the element + config: + + # Override autotools element default configure-commands + configure-commands: + - "%{configure} --enable-fancy-feature" + + # Specify public domain data, visible to other elements. + public: + bst: + integration-commands: + - /usr/bin/update-fancy-feature-cache + + # Specify a user id and group id to use in the build sandbox. + sandbox: + build-uid: 0 + build-gid: 0 + + +For most use cases you would not need to specify this much detail, we've provided +details here in order to have a more complete initial example. + +Let's break down the above and give a brief explanation of what these attributes mean. + + +.. _format_element_names: + +Element names and paths +~~~~~~~~~~~~~~~~~~~~~~~ +An *element name* is the filename of an element relative to the project's +:ref:`element path `. + +Element names are the identifiers used to refer to elements, they are used +to specify an element's :ref:`dependencies `, to select +elements to build on the :ref:`command line `, and they are arbitrarily +used in various element specific configuration surfaces, for example the +*target* configuration of the :mod:`link ` element is also +an *element name*. + + +Addressing elements +''''''''''''''''''' +When addressing elements in a single project, it is sufficient to use +the *element name* as a dependency or configuration parameter. + +When muliple projects are connected through :mod:`junction ` +elements, there is a need to address elements which are not in the same +project but in a junctioned *subproject*. In the case that you need to +address elements across junction boundaries, one must use *element paths*. + +An *element path* is a path to the element indicating the junction +elements leading up to the project, separated by ``:`` symbols, e.g.: +``junction.bst:element.bst``. + +Elements can be address across multiple junction boundaries with multiple +``:`` separators, e.g.: ``junction.bst:junction.bst:element.bst``. + + +Element naming rules +'''''''''''''''''''' +When naming the elements, use the following rules: + +* The name of the file must have the ``.bst`` extension. + +* All characters in the name must be printable 7-bit ASCII characters. + +* Following characters are reserved and must not be part of the name: + + - ``<`` (less than) + - ``>`` (greater than) + - ``:`` (colon) + - ``"`` (double quote) + - ``/`` (forward slash) + - ``\`` (backslash) + - ``|`` (vertical bar) + - ``?`` (question mark) + - ``*`` (asterisk) + + +Kind +~~~~ + +.. code:: yaml + + # Specify the kind of element this is + kind: autotools + +The ``kind`` attribute specifies which plugin will be operating on the element's input to +produce its output. Plugins define element types and each of them can be referred to by +name with the ``kind`` attribute. + +To refer to a third party plugin, prefix the plugin with its package, for example: + +.. code:: yaml + + kind: buildstream-plugins:dpkg_build + + +.. _format_depends: + +Depends +~~~~~~~ + +.. code:: yaml + + # Specify some dependencies + depends: + - element1.bst + - element2.bst + +Relationships between elements are specified with the ``depends`` attribute. Elements +may depend on other elements by specifying the :ref:`element names ` +they depend on here. + +See :ref:`format_dependencies` for more information on the dependency model. + + +.. _format_build_depends: + +Build-Depends +~~~~~~~~~~~~~ + +.. code:: yaml + + # Specify some build-dependencies + build-depends: + - element1.bst + - element2.bst + +Build dependencies between elements can be specified with the ``build-depends`` attribute. +The above code snippet is equivalent to: + +.. code:: yaml + + # Specify some build-dependencies + depends: + - filename: element1.bst + type: build + - filename: element2.bst + type: build + +See :ref:`format_dependencies` for more information on the dependency model. + + +.. _format_runtime_depends: + +Runtime-Depends +~~~~~~~~~~~~~~~ + +.. code:: yaml + + # Specify some runtime-dependencies + runtime-depends: + - element1.bst + - element2.bst + +Runtime dependencies between elements can be specified with the ``runtime-depends`` attribute. +The above code snippet is equivalent to: + +.. code:: yaml + + # Specify some runtime-dependencies + depends: + - filename: element1.bst + type: runtime + - filename: element2.bst + type: runtime + +See :ref:`format_dependencies` for more information on the dependency model. + + +.. _format_sources: + +Sources +~~~~~~~ + +.. code:: yaml + + # Specify the source which should be built + sources: + - kind: git + url: upstream:modulename.git + track: master + ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 + +Here we specify some input for the element, any number of sources may be specified. +By default the sources will be staged in the root of the element's build directory +in the build sandbox, but sources may specify a ``directory`` attribute to control +where the sources will be staged. The ``directory`` attribute may specify a build +sandbox relative subdirectory. + +For example, one might encounter a component which requires a separate data package +in order to build itself, in this case the sources might be listed as: + +.. code:: yaml + + sources: + + # Specify the source which should be built + - kind: git + url: upstream:modulename.git + track: master + ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 + + # Specify the data package we need for build frobnication, + # we need it to be unpacked in a src/frobdir + - kind: tarball + directory: src/frobdir + url: data:frobs.tgz + ref: 9d4b1147f8cf244b0002ba74bfb0b8dfb3... + +Like Elements, Source types are plugins which are indicated by the ``kind`` attribute. +Asides from the common ``kind`` and ``directory`` attributes which may be applied to all +Sources, refer to the Source specific documentation for meaningful attributes for the +particular Source. + + +Variables +~~~~~~~~~ + +.. code:: yaml + + # Override some variables + variables: + sysconfdir: "%{prefix}/etc" + +Variables can be declared or overridden from an element. Variables can also be +declared and overridden in the :ref:`projectconf` + +See :ref:`format_variables` below for a more in depth discussion on variables in BuildStream. + + +.. _format_environment: + +Environment +~~~~~~~~~~~ + +.. code:: yaml + + # Tweak the sandbox shell environment + environment: + LD_LIBRARY_PATH: /some/custom/path + +Environment variables can be set to literal values here, these environment +variables will be effective in the :mod:`Sandbox ` where +build instructions are run for this element. + +Environment variables can also be declared and overridden in the :ref:`projectconf` + + +.. _format_config: + +Config +~~~~~~ + +.. code:: yaml + + # Specify the configuration of the element + config: + + # Override autotools element default configure-commands + configure-commands: + - "%{configure} --enable-fancy-feature" + +Here we configure the element itself. The autotools element provides sane defaults for +building sources which use autotools. Element default configurations can be overridden +in the ``project.conf`` file and additionally overridden in the declaration of an element. + +For meaningful documentation on what can be specified in the ``config`` section for a given +element ``kind``, refer to the :ref:`element specific documentation `. + + +.. _format_public: + +Public +~~~~~~ + +.. code:: yaml + + # Specify public domain data, visible to other elements. + public: + bst: + integration-commands: + - /usr/bin/update-fancy-feature-cache + +Metadata declared in the ``public`` section of an element is visible to +any other element which depends on the declaring element in a given pipeline. +BuildStream itself consumes public data from the ``bst`` domain. The ``integration-commands`` +demonstrated above for example, describe commands which should be run in an +environment where the given element is installed but before anything should be run. + +An element is allowed to read domain data from any element it depends on, and users +may specify additional domains to be understood and processed by their own element +plugins. + +The public data keys which are recognized under the ``bst`` domain +can be viewed in detail in the :ref:`builtin public data ` section. + + +.. _format_sandbox: + +Sandbox +~~~~~~~ +Configuration for the build sandbox (other than :ref:`environment variables `) +can be placed in the ``sandbox`` configuration. The UID and GID used by the user +in the group can be specified, as well as the desired OS and machine +architecture. Possible machine architecture follow the same list as specified in +the :ref:`architecture option `. + +.. code:: yaml + + # Specify a user id and group id to use in the build sandbox. + sandbox: + build-uid: 1003 + build-gid: 1001 + +BuildStream normally uses uid 0 and gid 0 (root) to perform all +builds. However, the behaviour of certain tools depends on user id, +behaving differently when run as non-root. To support those builds, +you can supply a different uid or gid for the sandbox. Only +bwrap-style sandboxes support custom user IDs at the moment, and hence +this will only work on Linux host platforms. + +.. code:: yaml + + # Specify build OS and architecture + sandbox: + build-os: AIX + build-arch: power-isa-be + +When building locally, if these don't match the host machine then generally the +build will fail. The exception is when the OS is Linux and the architecture +specifies an ``x86-32`` build on an ``x86-64`` machine, or ``aarch32`` build on +a ``aarch64`` machine, in which case the ``linux32`` command is prepended to the +bubblewrap command. + +When building remotely, the OS and architecture are added to the ``Platform`` +field in the ``Command`` uploaded. Whether this actually results in a building +the element for the desired OS and architecture is dependent on the server +having implemented these options the same as buildstream. + + +.. _format_dependencies: + +Dependencies +------------ +The dependency model in BuildStream is simplified by treating software distribution +and software building as separate problem spaces. This is to say that one element +can only ever depend on another element but never on a subset of the product which +another element produces. + +In this section we'll quickly go over the few features BuildStream offers in its +dependency model. + + +Expressing dependencies +~~~~~~~~~~~~~~~~~~~~~~~ +Dependencies in BuildStream are parameterizable objects, however as demonstrated +in the :ref:`above example `, they can also be expressed as simple +strings as a convenience shorthand in most cases, whenever the default dependency +attributes are suitable. + +.. note:: + + Note the order in which element dependencies are declared in the ``depends``, + ``build-depends`` and ``runtime-depends`` lists are not meaningful. + +Dependency dictionary: + +.. code:: yaml + + # Fully specified dependency + depends: + - filename: foo.bst + type: build + junction: baseproject.bst + strict: false + +Attributes: + +* ``filename`` + + The :ref:`element name ` to depend on, or a list of mutiple element names. + + Specifying multiple element names in a single dependency will result in multiple dependencies + being declared with common properties. + + For example, one can declare multiple build dependencies with the same junction: + + .. code:: yaml + + # Declare three build dependencies from subproject.bst + depends: + - type: build + junction: subproject.bst + filename: + - element-a.bst + - element-b.bst + - element-c.bst + +* ``junction`` + + This attribute can be used to specify the junction portion of the :ref:`element name ` + separately from the project local element name. + + This should be the *element name* of the :mod:`junction ` element + in the local project, possibly followed by other junctions in subprojects leading + to the project in which the element you want to depend on resides. + + In the case that a *junction* is specified, the ``filename`` attribute indicates an + element in the *junctioned project*. + +* ``type`` + + This attribute is used to express the :ref:`dependency type `. + This field is not permitted in the :ref:`build-depends ` or + :ref:`runtime-depends ` lists. + +* ``strict`` + + This attribute can be used to specify that this element should + be rebuilt when the dependency changes, even when + :ref:`strict mode ` has been turned off. + + This is appropriate whenever a dependency's output is consumed + verbatim in the output of the depending element, for instance + when static linking is in use. + +* ``config`` + + This attribute defines the custom :term:`dependency configuration `, + which is supported by select :mod:`Element ` implementations. + + Elements which support :term:`dependency configuration ` do so + by implementing the + :func:`Element.configure_dependencies() ` + abstract method. It is up to each element or abstract element class to + document what is supported in their :term:`dependency configuration `. + + .. attention:: + + It is illegal to declare :term:`dependency configuration ` + on runtime dependencies, since runtime dependencies are not visible to the depending + element. + + +Redundant dependency declarations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +It is permitted to declare dependencies multiple times on the same element in the same +element declaration, the result will be an inclusive OR of all configurations you have +expressed in the redundant dependencies on the same element. + +* If a dependency is defined once as a ``build`` dependency and once as a ``runtime`` + :ref:`dependency type `, then the resulting dependency + type will be ``all`` + +* If any of the redundantly declared dependencies are specified as ``strict``, then + the resulting dependency will be ``strict``. + +Declaring redundant dependencies on the same element can be interesting when you +need to specify multiple :term:`dependency configurations ` +for the same element. For example, one might want to stage the same dependency +in multiple locations in the build sandbox. + + +Cross-junction dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +As explained in the :ref:`element name ` section +on element addressing, elements can be addressed across junction boundaries +using *element paths* such as ``junction.bst:element.bst``. An element +at any depth can be specified by specifying multiple junction elements. + +For example, one can specify a subproject element dependency with +the following syntax: + +.. code:: yaml + + build-depends: + - baseproject.bst:element.bst + +And one can specify an element residing in a sub-subproject as a +dependency like so: + +.. code:: yaml + + depends: + - baseproject.bst:middleproject.bst:element.bst + + +.. _format_dependencies_types: + +Dependency types +~~~~~~~~~~~~~~~~ +The dependency ``type`` attribute defines what the dependency is required for +and is essential to how BuildStream plots a build plan. + +There are three types which one can specify for a dependency: + +* ``build`` + + A ``build`` dependency type states that the given element's product must + be staged in order to build the depending element. Depending on an element + which has ``build`` dependencies will not implicitly depend on that element's + ``build`` dependencies. + + For convenience, these can be specified under the :ref:`build-depends ` + list. + +* ``runtime`` + + A ``runtime`` dependency type states that the given element's product + must be present for the depending element to function. An element's + ``runtime`` dependencies are not available to the element at build time. + + For convenience, these can be specified under the :ref:`runtime-depends ` + list. + +* ``all`` + + An ``all`` dependency is the default dependency type. If ``all`` is specified, + or if ``type`` is not specified at all, then it is assumed that the dependency + is required both at build time and runtime. + +.. note:: + + It is assumed that a dependency which is required for building an + element must run while building the depending element. This means that + ``build`` depending on a given element implies that that element's + ``runtime`` dependencies will also be staged for the purpose of building. + + +.. _format_variables: + +Using variables +--------------- +Variables in BuildStream are a way to make your build instructions and +element configurations more dynamic. + + +Referring to variables +~~~~~~~~~~~~~~~~~~~~~~ +Variables are expressed as ``%{...}``, where ``...`` must contain only +alphanumeric characters and the separators ``_`` and ``-``. Further, the +first letter of ``...`` must be an alphabetic character. + +.. code:: yaml + + This is release version %{version} + + +Declaring and overriding variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To declare or override a variable, one need only specify a value +in the relevant *variables* section: + +.. code:: yaml + + variables: + hello: Hello World + +You can refer to another variable while declaring a variable: + +.. code:: yaml + + variables: + release-text: This is release version %{version} + +The order in which you declare variables is arbitrary, so long as there is no cyclic +dependency and that all referenced variables are declared, the following is fine: + +.. code:: yaml + + variables: + release-text: This is release version %{version} + version: 5.5 + +.. note:: + + It should be noted that variable resolution only happens after all + :ref:`Element Composition ` has already taken place. + + This is to say that overriding ``%{version}`` at a higher priority will affect + the final result of ``%{release-text}``. + + +**Example:** + +.. code:: yaml + + kind: autotools + + # Declare variable, expect %{version} was already declared + variables: + release-text: This is release version %{version} + + config: + + # Customize the installation + install-commands: + - | + %{make-install} RELEASE_TEXT="%{release-text}" + + +Variables declared by BuildStream +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +BuildStream declares a set of :ref:`builtin ` +variables that may be overridden. In addition, the following +read-only variables are also dynamically declared by BuildStream: + +* ``element-name`` + + The name of the element being processed (e.g base/alpine.bst). + +* ``project-name`` + + The name of project where BuildStream is being used. + +* ``project-root`` & ``project-root-uri`` + + The directory where the project is located on the host. + + This variable is only available when declaring + :ref:`source alias values ` or + :ref:`source mirror values ` and allows + access to files in a project on the build host. + + * The ``project-root`` variable is a regular absolute path + * The ``project-root-uri`` variable is a properly quoted ``file://`` URI + + .. tip:: + + Use this variable to declare :ref:`source alias values ` + to refer to files which you store as a part of your project, e.g. tarballs + which you have committed to you BuildStream project. + + .. attention:: + + This feature has been provided for convenience when putting together a + project without the use of proper infrastructure. + + A better long term solution for accessing internal binaries and source + code is to setup internal infrastructure in your organization and use + the regular ways to access these sources from a well known internal URI. + +* ``toplevel-root`` & ``toplevel-root-uri`` + + The directory where the toplevel project is located on the host. + + This variable is only available when declaring + :ref:`source alias values ` or + :ref:`source mirror values ` and allows + access to files in a project on the build host. + + * The ``toplevel-root`` variable is a regular absolute path + * The ``toplevel-root-uri`` variable is a properly quoted ``file://`` URI + + .. tip:: + + Use this variable to declare :ref:`source alias values ` + to refer to files which you do not store as a part of your project, e.g. + tarballs or git repositories which must be placed in a directory within + the toplevel project before running the build. + + .. attention:: + + This feature has been provided for convenience when putting together a + project without the use of proper infrastructure. + + A better long term solution for accessing internal binaries and source + code is to setup internal infrastructure in your organization and use + the regular ways to access these sources from a well known internal URI. + +* ``max-jobs`` + + Maximum number of parallel build processes within a given + build, support for this is conditional on the element type + and the build system used (any element using 'make' can + implement this). diff --git a/_sources/format_intro.rst.txt b/_sources/format_intro.rst.txt new file mode 100644 index 000000000..9c19225d4 --- /dev/null +++ b/_sources/format_intro.rst.txt @@ -0,0 +1,358 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Introduction +============ +At the core of BuildStream is a data model of :mod:`Elements ` which +are parsed from ``.bst`` files in a project directory and configured from a few different +sources. + +When BuildStream loads your project, various levels of composition occur, allowing +configuration on various levels with different priority. + +This page provides an introduction to the project directory structure, explains the +basic *directives* supported inherently throughout the format, and outlines how composition +occurs and what configurations are considered in which order. + +The meaning of the various constructs expressed in the BuildStream format are covered +in other sections of the documentation. + +.. _format_structure: + + +Directory structure +------------------- +A BuildStream project is a directory consisting of: + +* A project configuration file +* BuildStream element files +* Optional user defined plugins +* An optional project.refs file + +A typical project structure may look like this:: + + myproject/project.conf + myproject/project.refs + myproject/elements/element1.bst + myproject/elements/element2.bst + myproject/elements/... + myproject/plugins/customelement.py + myproject/plugins/customelement.yaml + myproject/plugins/... + + +Except for the project configuration file, the user is allowed to structure +their project directory in any way. For documentation on the format of the project +configuration file, refer to the :ref:`projectconf` documentation. + +Simpler projects may choose to place all element definition files at the +root of the project directory while more complex projects may decide to +put stacks in one directory and other floating elements into other directories, +perhaps placing deployment elements in another directory, this is all fine. + +The important part to remember is that when you declare dependency relationships, +a project relative path to the element one depends on must be provided. + + +.. _format_composition: + +Composition +----------- +Below are the various sources of configuration which go into an element or source in the +order in which they are applied. Configurations which are applied later have a higher +priority and override configurations which precede them. + + +1. Builtin defaults +~~~~~~~~~~~~~~~~~~~ +The :ref:`builtin defaults ` provide a set of builtin +default default values for ``project.conf``. + +The project wide defaults defined in the builtin project configuration, such as the +*variables* or *environment* sections, form the base configuration of all elements. + + +2. Project configuration +~~~~~~~~~~~~~~~~~~~~~~~~ +The :ref:`project wide defaults ` specified in your +``project.conf`` are now applied on top of builtin defaults. + +Defaults such as the :ref:`variables ` or +:ref:`environment ` which are specified in +your ``project.conf`` override the builtin defaults for elements. + +Note that :ref:`plugin type specific configuration ` +in ``project.conf`` is not applied until later. + + +3. Plugin defaults +~~~~~~~~~~~~~~~~~~ +Elements and Sources are all implemented as plugins. + +Each Element plugin installs a ``.yaml`` file along side their plugin to +define the default *variables*, *environment* and *config*. The *config* +is element specific and as such this is the first place where defaults +can be set on the *config* section. + +The *variables* and *environment* specified in the declaring plugin's +defaults here override the project configuration defaults for the given +element ``kind``. + +Source plugins do not have a ``.yaml`` file, and do not have *variables* or +*environment*. + + +4. Project configuration overrides +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The ``project.conf`` now gives you :ref:`another opportunity ` to +override configuration on a per plugin basis. + +Configurations specified in the :ref:`elements ` or +:ref:`sources ` sections of the ``project.conf`` +will override the given plugin's defaults. + +In this phase, it is possible to override any configurations of a given plugin, +including configuration in element specific *config* sections. + +See also :ref:`project_overrides` + + +5. Plugin declarations +~~~~~~~~~~~~~~~~~~~~~~~ +Finally, after having resolved any :ref:`conditionals ` +in the parsing phase of loading element declarations; the configurations specified in a +``.bst`` file have the last word on any configuration in the data model. + + +.. _format_directives: + +Directives +---------- + +.. _format_directives_conditional: + +(?) Conditionals +~~~~~~~~~~~~~~~~ +The ``(?)`` directive allows expression of conditional statements which +test :ref:`project option ` values. + +The ``(?)`` directive may appear as a key in any dictionary expressed +in YAML, and its value is a list of conditional expressions. Each conditional +expression must be a single key dictionary, where the key is the conditional +expression itself, and the value is a dictionary to be composited into the +parent dictionary containing the ``(?)`` directive if the expression evaluates +to a truthy value. + +**Example:** + +.. code:: yaml + + variables: + prefix: "/usr" + enable-debug: False + (?): + - relocate == True: + prefix: "/opt" + - debug == True: + enable-debug: True + + +Expressions are evaluated in the specified order, and each time an expression +evaluates to a truthy value, its value will be composited to the parent dictionary +in advance of processing other elements, allowing for logically overriding previous +decisions in the condition list. + +Nesting of conditional statements is also supported. + +**Example:** + +.. code:: yaml + + variables: + enable-logging: False + enable-debug: False + (?): + - logging == True: + enable-logging: True + (?): + - debugging == True: + enable-debug: True + + +Conditionals are expressed in a pythonic syntax, the specifics for +testing the individually supported option types are described in +their :ref:`respective documentation `. + +Compound conditionals are also allowed. + +**Example:** + +.. code:: yaml + + variables: + enable-debug: False + (?): + - (logging == True and debugging == True): + enable-debug: True + +.. important:: + + Conditional statements are guaranteed to always be resolved in the + context of the project where the conditional statement is *declared*. + + When :ref:`including a file ` from a + subproject, any conditionals expressed in that file will already be + resolved in the context of the subproject which the file was included + from. + + +.. _format_directives_assertion: + +(!) Assertions +~~~~~~~~~~~~~~ +Assertions allow the project author to abort processing and present +a custom error message to the user building their project. + +This is only useful when used with conditionals, allowing the project +author to assert some invalid configurations. + + +**Example:** + +.. code:: yaml + + variables: + (?): + - (logging == False and debugging == True): + + (!): | + + Impossible to print any debugging information when + logging is disabled. + + +.. _format_directives_list_prepend: + +(<) List Prepend +~~~~~~~~~~~~~~~~ +Indicates that the list should be prepended to the target list, +instead of the default behavior which is to replace the target list. + +**Example:** + +.. code:: yaml + + config: + configure-commands: + # Before configuring, lets make sure we're using + # the latest config.sub & config.guess + (<): + - cp %{datadir}/automake-*/config.{sub,guess} . + + +.. _format_directives_list_append: + +(>) List Append +~~~~~~~~~~~~~~~ +Indicates that the list should be appended to the target list, instead +of the default behavior which is to replace the target list. + +**Example:** + +.. code:: yaml + + public: + bst: + split-rules: + devel: + # This element also adds some extra stubs which + # need to be included in the devel domain + (>): + - "%{libdir}/*.stub" + + +.. _format_directives_list_overwrite: + +(=) List Overwrite +~~~~~~~~~~~~~~~~~~ +Indicates that the list should be overwritten completely. + +This exists mostly for completeness, and we recommend using literal +lists most of the time instead of list overwrite directives when the +intent is to overwrite a list. + +This has the same behavior as a literal list, except that an +error will be triggered in the case that there is no underlying +list to overwrite; whereas a literal list will simply create a new +list. + +The added error protection can be useful when intentionally +overwriting a list in an element's *public data*, which is mostly +free form and not validated. + + +**Example:** + +.. code:: yaml + + config: + install-commands: + # This element's `make install` is broken, replace it. + (=): + - cp src/program %{bindir} + + +.. _format_directives_include: + +(@) Include +~~~~~~~~~~~ +Indicates that content should be loaded from files. + +The include directive expects a string, or a list of strings when +including multiple files. Each of these strings represent a project +relative filename to include. Files can be included from subprojects +by prefixing the string with the locally defined :mod:`junction +element ` and colon (':'). + +The include directive can be used in any dictionary declared in the +:ref:`project.conf `, in any :ref:`.bst file +`, or recursively included in another include file. + +The including YAML fragment has priority over the files it includes, +and overrides any values introduced by the includes. When including +multiple files, files are included in the order they are declared in +the include list, and each subsequent include file takes priority over +the previous one. + +**Example:** + +.. code:: yaml + + environment: + (@): junction.bst:includes/environment.bst + +.. important:: + + Files included across a junction cannot be used to inform the + declaration of a :mod:`junction element `, as + this can present a circular dependency. + + Any :ref:`variables `, :ref:`element + overrides `, :ref:`source + overrides ` or :ref:`mirrors + ` used in the declaration of a junction + must be declared in the :ref:`project.conf ` or in + included files which are local to the project declaring the + junction itself. diff --git a/_sources/format_project.rst.txt b/_sources/format_project.rst.txt new file mode 100644 index 000000000..cc56e4b35 --- /dev/null +++ b/_sources/format_project.rst.txt @@ -0,0 +1,1344 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _projectconf: + + +Project configuration +===================== +The project configuration file should be named ``project.conf`` and +be located at the project root. It holds information such as Source +aliases relevant for the sources used in the given project as well as +overrides for the configuration of element types used in the project. + +Values specified in the project configuration override any of the +default BuildStream project configuration, which is included +:ref:`here ` for reference. + + +.. _project_essentials: + +Essentials +---------- + + +.. _project_format_name: + +Project name +~~~~~~~~~~~~ +The project name is a unique symbol for your project and will +be used to distinguish your project from others in user preferences, +namespacing of your project's artifacts in shared artifact caches, +and in any case where BuildStream needs to distinguish between multiple +projects. + +The first thing to setup in your ``project.conf`` should be the name +of your project. + +.. code:: yaml + + name: my-project-name + +The project name may contain alphanumeric characters, dashes and +underscores, and may not start with a leading digit. + +.. attention:: + + The project name must be specified in the ``project.conf`` and + cannot be :ref:`included ` from a separate file. + + +.. _project_min_version: + +Minimum version +~~~~~~~~~~~~~~~ +The BuildStream format is guaranteed to be backwards compatible +with any earlier minor point releases, which is to say that +BuildStream 1.4 can read projects written for BuildStream 1.0, +and that BuildStream 2.2 can read projects written for BuildStream 2.0. + +Projects are required to specify the minimum version of BuildStream +which it requires, this allows project authors to convey a useful +error message to their users and peers, in the case that a user needs +to get a newer version of BuildStream in order to work with a given +project. + +The project's minimum required BuildStream version must be specified +in ``project.conf`` using the ``min-version`` field, e.g.: + +.. code:: yaml + + # This project uses features which were added in 2.2 + min-version: 2.2 + +It is recommended that when using new features, always consult this +documentation and observe which BuildStream version a feature you are +using was added in. If a feature in the BuildStream YAML format is +not documented with a specific *Since* version, you can assume that +it has been there from the beginning. + + +.. note:: + + External :mod:`Element ` and :mod:`Source ` + plugins also implement their own YAML configuration fragments and as + such are revisioned separately from the core format. + +.. attention:: + + The ``min-version`` must be specified in the ``project.conf`` and + cannot be :ref:`included ` from a separate file. + + +.. _project_element_path: + +Element path +~~~~~~~~~~~~ +To allow the user to structure their project nicely, BuildStream +allows the user to specify a project subdirectory where element +``.bst`` files are stored. + +.. code:: yaml + + element-path: elements + +Note that elements are referred to by their relative paths, whenever +elements are referred to in a ``.bst`` file or on the command line. + +.. attention:: + + The ``element-path`` can only be specified in the ``project.conf`` and + cannot be :ref:`included ` from a separate file. + + +.. _project_format_ref_storage: + +Ref storage +~~~~~~~~~~~ +By default, BuildStream expects to read and write source references +directly in the :ref:`source declaration `, but this +can be inconvenient and prohibitive in some workflows. + +Alternatively, BuildStream allows source references to be stored +centrally in a :ref:`project.refs file ` in the toplevel +:ref:`project directory `. + +This can be controlled with the ``ref-storage`` option, which is +allowed to be configured with the following values: + +* ``inline`` + + Source references are stored directly in the + :ref:`source declaration ` + +* ``project.refs`` + + Source references are stored in the ``project.refs`` file, and + junction source references are stored in the ``junction.refs`` file. + +To enable storing of source references in ``project.refs``, add the +following to your ``project.conf``: + +.. code:: yaml + + ref-storage: project.refs + +.. attention:: + + **Storing subproject source references in project.refs** + + When using the ``project.refs`` file, it is possible to override the + references in subprojects by editing the ``project.refs`` file directly + or by using :ref:`bst source track --cross-junctions `, + this can be practical to try out fresher versions of components which + are maintained in a subproject. + + It should be noted however that overridden subproject source references listed + in your ``project.refs`` file will be ignored by projects which use your project + as a subproject. + + +.. _configurable_warnings: + +Configurable Warnings +~~~~~~~~~~~~~~~~~~~~~ +Warnings can be configured as fatal using the ``fatal-warnings`` configuration item. +When a warning is configured as fatal, where a warning would usually be thrown instead an error will be thrown +causing the build to fail. + +Individual warnings can be configured as fatal by setting ``fatal-warnings`` to a list of warnings. + +.. code:: yaml + + fatal-warnings: + - overlaps + - ref-not-in-track + - : + +BuildStream provides a collection of :class:`Core Warnings ` which may be raised +by a variety of plugins. Other configurable warnings are plugin specific and should be noted within their individual documentation. + + +.. _project_source_aliases: + +Source aliases +~~~~~~~~~~~~~~ +In order to abstract the download location of source code and +any assets which need to be downloaded, and also as a matter of +convenience, BuildStream allows one to create named aliases for +URLs which are to be used in the individual ``.bst`` files. + +.. code:: yaml + + aliases: + foo: git://git.foo.org/ + bar: http://bar.com/downloads/ + +If you want this project's alias definitions to also be used for subprojects, +see :ref:`Mapping source aliases of subprojects `. + + +Sandbox options +~~~~~~~~~~~~~~~ +Sandbox options for the whole project can be supplied in +``project.conf`` in the same way as in an element. See :ref:`element configuration ` +for more detail. + +.. code:: yaml + + # Specify a user id and group id to use in the build sandbox. + sandbox: + build-uid: 1003 + build-gid: 1001 + + +.. _project_artifact_cache: + +Artifact server +~~~~~~~~~~~~~~~ +When maintaining a BuildStream project, it can be convenient to downstream users +of your project to provide access to a :ref:`cache server ` you maintain. + +The project can provide *recommended* artifact cache servers through project configuration +using the same semantics as one normally uses in the ``servers`` list of the +:ref:`cache server user configuration `: + +.. code:: yaml + + # + # A remote cache from which to download prebuilt artifacts + # + artifacts: + - url: https://foo.com:11001 + auth: + server-cert: server.crt + +.. attention:: + + Unlike user configuration, the filenames provided in the :ref:`auth ` + configuration block are relative to the :ref:`project directory `. + + It is recommended to include public keys such as the ``server-cert`` along with your + project so that downstream users can have automatic read access to your project. + + To provide write access to downstream users, it is recommended that the required + private keys such as the ``client-key`` be provided to users out of band, + and require that users configure write access separately in their own + :ref:`user configuration `. + + +.. _project_source_cache: + +Source cache server +~~~~~~~~~~~~~~~~~~~ +In the same way as artifact cache servers, the project can provide *recommended* source cache +servers through project configuration using the same semantics as one normally uses in the +``servers`` list of the :ref:`cache server user configuration `: + +.. code:: yaml + + # + # A remote cache from which to download prestaged sources + # + source-caches: + - url: https://foo.com:11001 + auth: + server-cert: server.crt + +.. attention:: + + Unlike user configuration, the filenames provided in the :ref:`auth ` + configuration block are relative to the :ref:`project directory `. + + It is recommended to include public keys such as the ``server-cert`` along with your + project so that downstream users can have automatic read access to your project. + + To provide write access to downstream users, it is recommended that the required + private keys such as the ``client-key`` be provided to users out of band, + and require that users configure write access separately in their own + :ref:`user configuration `. + + +.. _project_essentials_mirrors: + +Mirrors +~~~~~~~ +A list of mirrors can be defined that couple a location to a mapping of aliases to a +list of URIs, e.g. + +.. code:: yaml + + mirrors: + - name: middle-earth + aliases: + foo: + - http://www.middle-earth.com/foo/1 + - http://www.middle-earth.com/foo/2 + bar: + - http://www.middle-earth.com/bar/1 + - http://www.middle-earth.com/bar/2 + - name: oz + aliases: + foo: + - http://www.oz.com/foo + bar: + - http://www.oz.com/bar + +The order that the mirrors (and the URIs therein) are consulted is in the order +they are defined when fetching, and in reverse-order when tracking. + +The mirrors can be overridden on a per project basis using +:ref:`user configuration `. One can also specify which mirror should +be used first in the :ref:`user configuration `, or using +the :ref:`--default-mirror ` command-line argument. + +If you want this project's mirrors to also be used for subprojects, +see :ref:`Mapping source aliases of subprojects `. + + +.. _project_plugins: + +Loading plugins +--------------- +If your project makes use of any custom :mod:`Element ` or +:mod:`Source ` plugins, then the project must inform BuildStream +of the plugins it means to make use of and the origin from which they can be loaded. + +Note that plugins with the same name from different origins are not permitted. + +.. attention:: + + The plugins can only be specified in the ``project.conf`` and cannot be + :ref:`included ` from a separate file. + + +.. _project_plugins_local: + +Local plugins +~~~~~~~~~~~~~ +Local plugins are expected to be found in a subdirectory of the actual +BuildStream project. :mod:`Element ` and +:mod:`Source ` plugins should be stored in separate +directories to avoid namespace collisions, you can achieve this by +specifying a separate *origin* for sources and elements. + +.. code:: yaml + + plugins: + + - origin: local + path: plugins/sources + + # We want to use the `mysource` source plugin located in our + # project's `plugins/sources` subdirectory. + sources: + - mysource + +There is no strict versioning policy for plugins loaded from the local +origin because the plugin is provided with the project data and as such, +it is considered to be completely deterministic. + +Usually your project will be managed by a VCS like git, and any changes +to your local plugins may have an impact on your project, such as changes +to the artifact cache keys produced by elements which use these plugins. +Changes to plugins might provide new YAML configuration options, changes +in the semantics of existing configurations or even removal of existing +YAML configurations. + + +.. _project_plugins_pip: + +Pip plugins +~~~~~~~~~~~ +Plugins loaded from the ``pip`` origin are expected to be installed +separately on the host operating system using python's package management +system. + +.. code:: yaml + + plugins: + + - origin: pip + + # Specify the name of the python package containing + # the plugins we want to load. The name one would use + # on the `pip install` command line. + # + package-name: potato + + # We again must specify specifically which plugins we + # want loaded from this origin. + # + elements: + - starch + +Unlike local plugins, plugins loaded from the ``pip`` origin are +loaded from the active *python environment*, and as such you do not +usually have full control over the plugins your project uses unless +one uses strict :ref:`version constraints `. + +The official plugin packages maintained by the BuildStream community are +guaranteed to be fully API stable. If one chooses to load these plugins +from the ``pip`` origin, then it is recommended to use *minimal bound dependency* +:ref:`constraints ` when using +official plugin packages so as to be sure that you have access to all the +features you intend to use in your project. + + +.. _project_plugins_pip_version_constraints: + +Versioning constraints +'''''''''''''''''''''' +When loading plugins from the ``pip`` plugin origin, it is possible to +specify constraints on the versions of packages you want to load +your plugins from. + +The syntax for specifying versioning constraints is the same format supported by +the ``pip`` package manager. + +.. note:: + + In order to be certain that versioning constraints work properly, plugin + packages should be careful to adhere to `PEP 440, Version Identification and Dependency + Specification `_. + +Here are a couple of examples: + +**Specifying minimal bound dependencies**: + +.. code:: yaml + + plugins: + + - origin: pip + + # This project uses the API stable potato project and + # requires features from at least version 1.2 + # + package-name: potato>=1.2 + +**Specifying exact versions**: + +.. code:: yaml + + plugins: + + - origin: pip + + # This project requires plugins from the potato + # project at exactly version 1.2.3 + # + package-name: potato==1.2.3 + +**Specifying version constraints**: + +.. code:: yaml + + plugins: + + - origin: pip + + # This project requires plugins from the potato + # project from version 1.2.3 onward until 1.3. + # + package-name: potato>=1.2.3,<1.3 + +.. important:: + + **Unstable plugin packages** + + When using unstable plugins loaded from the ``pip`` origin, the installed + plugins can sometimes be incompatible with your project. + + **Use virtual environments** + + Your operating system's default python environment can only have one + version of a given package installed at a time, if you work on multiple + BuildStream projects on the same host, they may not agree on which versions + of plugins to use. + + In order to guarantee that you can use a specific version of a plugin, + you may need to install BuildStream into a `virtual environment + `_ in order to control which + python package versions are available when using your project. + + Follow `these instructions + `_ + to install BuildStream in a virtual environment. + + **Possible junction conflicts** + + If you have multiple projects which are connected through + :mod:`junction ` elements, these projects can disagree + on which version of a plugin is needed from the ``pip`` origin. + + Since only one version of a given plugin *package* can be installed + at a time in a given *python environment*, you must ensure that all + projects connected through :mod:`junction ` elements + agree on which versions of API unstable plugin packages to use. + + +.. _project_plugins_junction: + +Junction plugins +~~~~~~~~~~~~~~~~ +Junction plugins are loaded from another project which your project has a +:mod:`junction ` declaration for. Plugins are loaded directly +from the referenced project, the source and element plugins listed will simply +be loaded from the subproject regardless of how they were defined in that project. + +Plugins loaded from a junction might even come from another junction and +be *deeply nested*. + +.. code:: yaml + + plugins: + + - origin: junction + + # Specify the local junction name declared in your + # project as the origin from where to load plugins from. + # + junction: subproject-junction.bst + + # Here we want to get the `frobnicate` element + # from the subproject and use it in our project. + # + elements: + - frobnicate + +Plugins loaded across junction boundaries will be loaded in the +context of your project, and any default values set in the ``project.conf`` +of the junctioned project will be ignored when resolving the +defaults provided with element plugins. + +It is recommended to use :ref:`include directives ` +in the case that the referenced plugins from junctioned projects depend +on variables defined in the project they come from, in this way you can include +variables needed by your plugins into your own ``project.conf``. + +.. tip:: + + **Distributing plugins as projects** + + It is encouraged that people use BuildStream projects to distribute plugins + which are intended to be shared among projects, especially when these plugins + are not guaranteed to be completely API stable. This can still be done while + also distributing your plugins as :ref:`pip packages ` at + the same time. + + This can be achieved by simply creating a repository or tarball which + contains only the plugins you want to distribute, along with a ``project.conf`` + file declaring these plugins as :ref:`local plugins `. + + Using plugins which are distributed as local plugins in a BuildStream project + ensures that you always have full control over which exact plugin your + project is using at all times, without needing to store the plugin as a + :ref:`local plugin ` in your own project. + + +.. _project_plugins_deprecation: + +Suppressing deprecation warnings +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Plugins can be deprecated over time, and using deprecated plugins will +trigger a warning when loading elements and sources which use +deprecated plugin kinds. + +These deprecation warnings can be suppressed for the entire plugin +origin or on a per plugin kind basis. + +To suppress all deprecation warnings from the origin, set the +``allow-deprecated`` flag for the origin as follows: + +.. code:: yaml + + plugins: + + - origin: local + path: plugins/sources + + # Suppress deprecation warnings for any plugins loaded here + allow-deprecated: True + + sources: + - mysource + + +In order to suppress deprecation warnings for a single element or +source kind within an origin, you will have to use a dictionary +to declare the specific plugin kind and set the ``allow-deprecated`` flag +on that dictionary as follows: + +.. code:: yaml + + plugins: + + - origin: pip + package-name: potato + + # Here we use a dictionary to declare the "starch" + # element kind, and specify that it is allowed to + # be deprecated. + # + elements: + - kind: starch + allow-deprecated: True + + +.. _project_options: + +Options +------- +Options are how BuildStream projects can define parameters which +can be configured by users invoking BuildStream to build your project. + +Options are declared in the ``project.conf`` in the main ``options`` +dictionary. + +.. code:: yaml + + options: + debug: + type: bool + description: Whether to enable debugging + default: False + +Project options can be specified on the command line using +:ref:`bst --option ... ` + +.. note:: + + The name of the option may contain alphanumeric characters + underscores, and may not start with a leading digit. + + +Common properties +~~~~~~~~~~~~~~~~~ +All option types accept the following common attributes + +* ``type`` + + Indicates the type of option to declare + +* ``description`` + + A description of the meaning of the option + +* ``variable`` + + Optionally indicate a :ref:`variable ` name to + export the option to. A string form of the selected option will + be used to set the exported value. + + If used, this value will override any existing value for the + variable declared in ``project.conf``, and will be overridden in + the regular :ref:`composition order `. + + .. note:: + + The name of the variable to export may contain alphanumeric + characters, dashes, underscores, and may not start with a leading + digit. + + +.. _project_options_bool: + +Boolean +~~~~~~~ +The ``bool`` option type allows specifying boolean values which +can be cased in conditional expressions. + + +**Declaring** + +.. code:: yaml + + options: + debug: + type: bool + description: Whether to enable debugging + default: False + + +**Evaluating** + +Boolean options can be tested in expressions with equality tests: + +.. code:: yaml + + variables: + enable-debug: False + (?): + - debug == True: + enable-debug: True + +Or simply treated as truthy values: + +.. code:: yaml + + variables: + enable-debug: False + (?): + - debug: + enable-debug: True + + +**Exporting** + +When exporting boolean options as variables, a ``True`` option value +will be exported as ``1`` and a ``False`` option as ``0`` + + +.. _project_options_enum: + +Enumeration +~~~~~~~~~~~ +The ``enum`` option type allows specifying a string value +with a restricted set of possible values. + + +**Declaring** + +.. code:: yaml + + options: + loglevel: + type: enum + description: The logging level + values: + - debug + - info + - warning + default: info + + +**Evaluating** + +Enumeration options must be tested as strings in conditional +expressions: + +.. code:: yaml + + variables: + enable-debug: False + (?): + - loglevel == "debug": + enable-debug: True + + +**Exporting** + +When exporting enumeration options as variables, the value is +exported as a variable directly, as it is a simple string. + + +.. _project_options_flags: + +Flags +~~~~~ +The ``flags`` option type allows specifying a list of string +values with a restricted set of possible values. + +In contrast with the ``enum`` option type, the *default* value +need not be specified and will default to an empty set. + + +**Declaring** + +.. code:: yaml + + options: + logmask: + type: flags + description: The logging mask + values: + - debug + - info + - warning + default: + - info + + +**Evaluating** + +Options of type ``flags`` can be tested in conditional expressions using +a pythonic *in* syntax to test if an element is present in a set: + +.. code:: yaml + + variables: + enable-debug: False + (?): + - ("debug" in logmask): + enable-debug: True + + +**Exporting** + +When exporting flags options as variables, the value is +exported as a comma separated list of selected value strings. + + +.. _project_options_arch: + +Architecture +~~~~~~~~~~~~ +The ``arch`` option type is a special enumeration option which defaults via +`uname -m` results to the following list. + +* aarch32 +* aarch64 +* aarch64-be +* power-isa-be +* power-isa-le +* sparc-v9 +* x86-32 +* x86-64 + +The reason for this, opposed to using just `uname -m`, is that we want an +OS-independent list, as well as several results mapping to the same architecture +(e.g. i386, i486 etc. are all x86-32). It does not support assigning any default +in the project configuration. + +.. code:: yaml + + options: + machine_arch: + type: arch + description: The machine architecture + values: + - aarch32 + - aarch64 + - x86-32 + - x86-64 + + +Architecture options can be tested with the same expressions +as other Enumeration options. + + +.. _project_options_os: + +OS +~~ + +The ``os`` option type is a special enumeration option, which defaults to the +results of `uname -s`. It does not support assigning any default in the project +configuration. + +.. code:: yaml + + options: + machine_os: + type: os + description: The machine OS + values: + - Linux + - SunOS + - Darwin + - FreeBSD + +Os options can be tested with the same expressions as other Enumeration options. + + +.. _project_options_element_mask: + +Element mask +~~~~~~~~~~~~ +The ``element-mask`` option type is a special Flags option +which automatically allows only element names as values. + +.. code:: yaml + + options: + debug_elements: + type: element-mask + description: The elements to build in debug mode + +This can be convenient for automatically declaring an option +which might apply to any element, and can be tested with the +same syntax as other Flag options. + + +.. code:: yaml + + variables: + enable-debug: False + (?): + - ("element.bst" in debug_elements): + enable-debug: True + + +.. _project_junctions: + +Junctions +--------- +In this section of ``project.conf``, we can define the relationship a project +has with :mod:`junction ` elements in the same project, or +even in subprojects. + +Sometimes when your project has multiple :mod:`junction ` elements, +a situation can arise where you have multiple instances of the same project loaded +at the same time. In most cases, you will want to reconcile this conflict by ensuring +that your projects share the same junction. In order to reconcile conflicts by +ensuring nested junctions to the same project are shared, please refer to +:ref:`the documentation on nested junctions `. + +In some exceptional cases, it is entirely intentional and appropriate to use +the same project more than once in the same build pipeline. The attributes +in the ``junctions`` group here in ``project.conf`` provide some tools you can +use to explicitly allow the coexistence of the same project multiple times. + + +Duplicate junctions +~~~~~~~~~~~~~~~~~~~ +In the case that you are faced with an error due to subprojects sharing +a common sub-subproject, you can use the ``duplicates`` configuration +in order to allow the said project to be loaded twice. + +**Example**: + +.. code:: yaml + + junctions: + + duplicates: + + # Here we use the packaging tooling completely separately from + # the payload that we are packaging, they are never staged to + # the same location in a given sandbox, and as such we would + # prefer to allow the 'runtime' project to be loaded separately. + # + # This statement will ensure that loading the 'runtime' project + # from these two locations will not produce any errors. + # + runtime: + - payload.bst:runtime.bst + - packaging.bst:runtime.bst + +When considering duplicated projects in the same pipeline, all instances +of the said project need to be marked as ``duplicates`` in order to avoid +a *conflicting junction error* at load time. + +.. tip:: + + The declaration of ``duplicates`` is inherited by any dependant projects + which may later decide to depend on your project. + + If you depend on a project which itself has ``duplicates``, and you need + to duplicate it again, then you only need to declare the new duplicate, + you do not need to redeclare duplicates redundantly. + + +Internal junctions +~~~~~~~~~~~~~~~~~~ +Another way to avoid *conflicting junction errors* when you know that your +subproject should not conflict with other instances of the same subproject, +is to declare the said subproject as *internal*. + +**Example**: + +.. code:: yaml + + junctions: + + # Declare this subproject as "internal" because we know + # that we only use it for build dependencies, and as such + # we know that it cannot collide with elements in dependant + # projects. + # + internal: + - special-compiler.bst + +When compared to *duplicates* above, *internal* projects have the advantage +of never producing any *conflicting junction errors* in dependant projects +(reverse dependency projects). + +This approach is preferrable in cases where you know for sure that dependant +projects will not be depending directly on elements from your internal +subproject. + +.. attention:: + + Declaring a junction as *internal* is a promise that dependant projects + will not accrue runtime dependencies on elements in your *internal* subproject. + + +.. _project_junctions_source_aliases: + +Mapping source aliases of subprojects +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:mod:`junction ` elements allow source aliases of subprojects +to be mapped to aliases of the parent project. This makes it possible to control +the translation of aliases to URLs including mirror configuration across multiple +project levels. + +To ensure that there are mappings for all aliases of all subprojects, you can set the +``disallow-subproject-uris`` flag in the ``junctions`` group here in ``project.conf``. + +top-level + +.. code:: yaml + + junctions: + disallow-subproject-uris: True + +This will raise an error if an alias without a mapping is encountered. This flag +is applied recursively across all junctions. + +It also configures ``unaliased-url`` as a fatal warning in all subprojects to +ensure that the current project is in full control over all source URLs. +As the fatal warning configuration contributes to the cache key, this flag will +affect the cache key of subprojects that haven't already configured +``unaliased-url`` as a fatal warning. + + +.. _project_defaults: + +Element default configuration +----------------------------- +The ``project.conf`` plays a role in defining elements by +providing default values and also by overriding values declared +by plugins on a plugin wide basis. + +See the :ref:`composition ` documentation for +more detail on how elements are composed. + + +.. _project_defaults_variables: + +Variables +~~~~~~~~~ +The defaults for :ref:`Variables ` used in your +project is defined here. + +.. code:: yaml + + variables: + prefix: "/usr" + + +.. _project_defaults_environment: + +Environment +~~~~~~~~~~~ +The defaults environment for the build sandbox is defined here. + +.. code:: yaml + + environment: + PATH: /usr/bin:/bin:/usr/sbin:/sbin + +Additionally, the special ``environment-nocache`` list which specifies +which environment variables do not affect build output, and are thus +not considered in the calculation of artifact keys can be defined here. + +.. code:: yaml + + environment-nocache: + - MAXJOBS + +Note that the ``environment-nocache`` list only exists so that we can +control parameters such as ``make -j ${MAXJOBS}``, allowing us to control +the number of jobs for a given build without affecting the resulting +cache key. + + +.. _project_split_rules: + +Split rules +~~~~~~~~~~~ +The project wide :ref:`split rules ` defaults can +be specified here. + +.. code:: yaml + + split-rules: + devel: + - | + %{includedir} + - | + %{includedir}/** + - | + %{libdir}/lib*.a + - | + %{libdir}/lib*.la + + +.. _project_overrides: + +Overriding plugin defaults +-------------------------- +Base attributes declared by element and source plugins can be overridden +on a project wide basis. This section explains how to make project wide +statements which augment the configuration of an element or source plugin. + + +.. _project_element_overrides: + +Element overrides +~~~~~~~~~~~~~~~~~ +The elements dictionary can be used to override variables, environments +or plugin specific configuration data as shown below. + + +.. code:: yaml + + elements: + + # Override default values for all autotools elements + autotools: + + variables: + bindir: "%{prefix}/bin" + + config: + configure-commands: ... + + environment: + PKG_CONFIG_PATH=%{libdir}/pkgconfig + + +.. _project_source_overrides: + +Source overrides +~~~~~~~~~~~~~~~~ +The sources dictionary can be used to override source plugin +specific configuration data as shown below. + + +.. code:: yaml + + sources: + + # Override default values for all git sources + git: + + config: + checkout-submodules: False + + +.. _project_shell: + +Customizing the shell +--------------------- +Since BuildStream cannot know intimate details about your host or about +the nature of the runtime and software that you are building, the shell +environment for debugging and testing applications may need some help. + +The ``shell`` section allows some customization of the shell environment. + + +Interactive shell command +~~~~~~~~~~~~~~~~~~~~~~~~~ +By default, BuildStream will use ``sh -i`` when running an interactive +shell, unless a specific command is given to the ``bst shell`` command. + +BuildStream will automatically set a convenient prompt via the ``PS1`` +environment variable for interactive shells; which might be overwritten +depending on the shell you use in your runtime. + +If you are using ``bash``, we recommend the following configuration to +ensure that the customized prompt is not overwritten: + +.. code:: yaml + + shell: + + # Specify the command to run by default for interactive shells + command: [ 'bash', '--noprofile', '--norc', '-i' ] + + +Environment assignments +~~~~~~~~~~~~~~~~~~~~~~~ +In order to cooperate with your host environment, a debugging shell +sometimes needs to be configured with some extra knowledge inheriting +from your host environment. + +This can be achieved by setting up the shell ``environment`` configuration, +which is expressed as a dictionary very similar to the +:ref:`default environment `, except that it +supports host side environment variable expansion in values. + +For example, to share your host ``DISPLAY`` and ``DBUS_SESSION_BUS_ADDRESS`` +environments with debugging shells for your project, specify the following: + +.. code:: yaml + + shell: + + # Share some environment variables from the host environment + environment: + DISPLAY: '$DISPLAY' + DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS' + +Or, a more complex example is how one might share the host pulseaudio +server with a ``bst shell`` environment: + +.. code:: yaml + + shell: + + # Set some environment variables explicitly + environment: + PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native' + + +Host files +~~~~~~~~~~ +It can be useful to share some files on the host with a shell so that +it can integrate better with the host environment. + +The ``host-files`` configuration allows one to specify files and +directories on the host to be bind mounted into the sandbox. + +.. warning:: + + One should never mount directories where one expects to + find data and files which belong to the user, such as ``/home`` + on POSIX platforms. + + This is because the unsuspecting user may corrupt their own + files accidentally as a result. Instead users can use the + ``--mount`` option of ``bst shell`` to mount data into the shell. + + +The ``host-files`` configuration is an ordered list of *mount specifications*. + +Members of the list can be *fully specified* as a dictionary, or a simple +string can be used if only the defaults are required. + +The fully specified dictionary has the following members: + +* ``path`` + + The path inside the sandbox. This is the only mandatory + member of the mount specification. + +* ``host_path`` + + The host path to mount at ``path`` in the sandbox. This + will default to ``path`` if left unspecified. + +* ``optional`` + + Whether the mount should be considered optional. This + is ``False`` by default. + + +Here is an example of a *fully specified mount specification*: + +.. code:: yaml + + shell: + + # Mount an arbitrary resolv.conf from the host to + # /etc/resolv.conf in the sandbox, and avoid any + # warnings if the host resolv.conf doesnt exist. + host-files: + - host_path: '/usr/local/work/etc/resolv.conf' + path: '/etc/resolv.conf' + optional: True + +Here is an example of using *shorthand mount specifications*: + +.. code:: yaml + + shell: + + # Specify a list of files to mount in the sandbox + # directory from the host. + # + # If these do not exist on the host, a warning will + # be issued but the shell will still be launched. + host-files: + - '/etc/passwd' + - '/etc/group' + - '/etc/resolv.conf' + +Host side environment variable expansion is also supported: + +.. code:: yaml + + shell: + + # Mount a host side pulseaudio server socket into + # the shell environment at the same location. + host-files: + - '${XDG_RUNTIME_DIR}/pulse/native' + + +.. _project_default_targets: + +Default targets +--------------- +When running BuildStream commands from a project directory or subdirectory +without specifying any target elements on the command line, the default targets +of the project will be used. The default targets can be configured in the +``defaults`` section as follows: + +.. code:: yaml + + defaults: + + # List of default target elements + targets: + - app.bst + +If no default targets are configured in ``project.conf``, BuildStream commands +will default to all ``.bst`` files in the configured element path. + +Commands that cannot support junctions as target elements (``bst build``, +``bst artifact push``, and ``bst artifact pull``) ignore junctions in the list +of default targets. + +When running BuildStream commands from a workspace directory (that is not a +BuildStream project directory), project default targets are not used and the +workspace element will be used as the default target instead. + +``bst artifact checkout``, ``bst source checkout``, and ``bst shell`` are +currently limited to a single target element and due to this, they currently +do not use project default targets. However, they still use the workspace +element as default target when run from a workspace directory. + + +.. _project_builtin_defaults: + +Builtin defaults +---------------- +BuildStream defines some default values for convenience, the default +values overridden by your project's ``project.conf`` are presented here: + + .. literalinclude:: ../../src/buildstream/data/projectconfig.yaml + :language: yaml diff --git a/_sources/format_project_refs.rst.txt b/_sources/format_project_refs.rst.txt new file mode 100644 index 000000000..140e8e58d --- /dev/null +++ b/_sources/format_project_refs.rst.txt @@ -0,0 +1,91 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _projectrefs: + +The project.refs file +===================== +If one has elected to store source references in a single ``project.refs`` +file, then it will be stored at the toplevel of your project directory +adjacent to ``project.conf``. This can be configured in your project +using the :ref:`ref-storage configuration ` + +Sources for :mod:`junction ` elements are stored +separately in an adjacent ``junction.refs`` file of the same format. + + +.. _projectrefs_basics: + +Basic behavior +-------------- +When a ``project.refs`` file is in use, any source references found +in the :ref:`inline source declarations ` are considered +invalid and will be ignored, and a warning will be emitted for them. + +When ``bst source track`` is run for your project, the ``project.refs`` file +will be updated instead of the inline source declarations. In the absence +of a ``project.refs`` file, ``bst source track`` will create one automatically +with the tracking results. + +An interesting property of ``project.refs`` is that it allows for +*cross junction tracking*. This is to say that it is possible to override +the *ref* of a given source in a project that your project depends on via +a :mod:`junction `, without actually modifying the +junctioned project. + + +.. _projectrefs_format: + +Format +------ +The ``project.refs`` uses the same YAML format used throughout BuildStream, +and supports the same :ref:`directives ` which apply to +``project.conf`` and element declaration files (i.e. *element.bst* files). + +The ``project.refs`` file format itself is very simple, it contains a single ``projects`` +key at the toplevel, which is a dictionary of :ref:`project names `. +Each *project name* is a dictionary of *element names*, and each *element name* holds +a list of dictionaries corresponding to the element's :ref:`sources `. + + +**Example** + +.. code:: yaml + + # Main toplevel "projects" key + projects: + + # The local project's name is "core" + core: + + # A dictionary of element names + base/automake.bst: + + # A list of sources corresponding to the element + # in the same order in which they were declared. + # + # The values of this list are dictionaries of the + # symbolic "ref" portion understood by the given + # source plugin implementation. + # + - ref: af6ba39142220687c500f79b4aa2f181d9b24e4... + + # The "core" project depends on the "bootstrap" project, + # here we are allowed to override the refs for the projects + # we depend on through junctions. + bootstrap: + + zlib.bst: + - ref: 4ff941449631ace0d4d203e3483be9dbc9da4540... diff --git a/_sources/format_public.rst.txt b/_sources/format_public.rst.txt new file mode 100644 index 000000000..fc51681b3 --- /dev/null +++ b/_sources/format_public.rst.txt @@ -0,0 +1,118 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _public_builtin: + +Builtin public data +=================== +Elements can provide public data which can be read by other elements +later in the pipeline, the format for exposing public data on a given +element is :ref:`described here `. + +Any element may use public data for whatever purpose it wants, but +BuildStream has some built-in expectations of public data, which resides +completely in the ``bst`` domain. + +In this section we will describe the public data in the ``bst`` domain. + + +.. _public_integration: + +Integration commands +-------------------- + +.. code:: yaml + + # Specify some integration commands + public: + bst: + integration-commands: + - /usr/bin/update-fancy-feature-cache + +The built-in ``integration-commands`` list indicates that depending elements +should run this set of commands before expecting the staged runtime environment +to be functional. + +Typical cases for this include running ``ldconfig`` at the base of a pipeline, +or running commands to update various system caches. + +Integration commands of a given element are automatically run by the +:func:`Element.integrate() ` method +and are used by various plugins. + +Notably the :mod:`BuildElement ` derived classes +will always integrate the build dependencies after staging and before running +any build commands. + + +.. _public_split_rules: + +Split rules +----------- + +.. code:: yaml + + # Specify some split rules + public: + bst: + split-rules: + runtime: + - | + %{bindir}/* + - | + %{sbindir}/* + - | + %{libexecdir}/* + - | + %{libdir}/lib*.so* + +Split rules indicate how the output of an element can be categorized +into *domains*. + +The ``split-rules`` domains are used by the +:func:`Element.stage_artifact() ` +method when deciding what domains of an artifact should be staged. + +The strings listed in each domain are first substituted with the +:ref:`variables ` in context of the given element, and +then applied as a glob style match, as understood by +:func:`utils.glob() ` + +This is used for creating compositions with the :mod:`compose ` +element and can be used by other deployment related elements for the purpose of +splitting element artifacts into separate packages. + + +.. _public_overlap_whitelist: + +Overlap whitelist +----------------- + +The overlap whitelist indicates which files this element is allowed to overlap +over other elements when staged together with other elements. + +Each item in the overlap whitelist has substitutions applied from +:ref:`variables `, and is then applied as a glob-style match +(i.e. :func:`utils.glob() `). + +.. code:: yaml + + public: + bst: + overlap-whitelist: + - | + %{sysconfdir}/* + - | + /etc/fontcache diff --git a/_sources/hacking/coding_guidelines.rst.txt b/_sources/hacking/coding_guidelines.rst.txt new file mode 100644 index 000000000..eafb0c7d9 --- /dev/null +++ b/_sources/hacking/coding_guidelines.rst.txt @@ -0,0 +1,899 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _coding_guidelines: + +Coding guidelines +----------------- +This section discusses coding style and other guidelines for hacking +on BuildStream. This is important to read through for writing any non-trivial +patches and especially outlines what people should watch out for when +reviewing patches. + +Much of the rationale behind what is layed out in this section considers +good traceability of lines of code with *git blame*, overall sensible +modular structure, consistency in how we write code, and long term maintenance +in mind. + + +Approximate PEP-8 Style +~~~~~~~~~~~~~~~~~~~~~~~ +Python coding style for BuildStream is approximately `pep8 `_. + +The coding style is automatically enforced by `black `_. + +Formatting will be checked automatically when running the testsuite on CI. For +details on how to format your code locally, see :ref:`formatting code `. + + +.. _contributing_documenting_symbols: + +Documenting symbols +~~~~~~~~~~~~~~~~~~~ +In BuildStream, we maintain what we call a *"Public API Surface"* that +is guaranteed to be stable and unchanging across stable releases. The +symbols which fall into this special class are documented using Python's +standard *docstrings*, while all other internals of BuildStream are documented +with comments above the related symbol. + +When documenting the public API surface which is rendered in the reference +manual, we always mention the major version in which the API was introduced, +as shown in the examples below. If a public API exists without the *Since* +annotation, this is taken to mean that it was available since the first stable +major point release (e.g: 2.0). + +We also always ensure that the **public API** is entirely typed using type +annotations inline. + +The private API *can* be typed inline or in the documentation at the author's +discretion. + +.. note:: + + Types are checked using `mypy`. You can run it like :command:`tox -e mypy` + +Here are some examples to get the hang of the format of API documenting +comments and docstrings. + +**Public API Surface method**:: + + def frobnicate(self, source: Source, *, frobilicious: bool = False) -> Element: + """Frobnicates this element with the specified source + + Args: + source: The Source to frobnicate with + frobilicious: Optionally specify that frobnication should be + performed frobiliciously + + Returns: + The frobnicated version of this Element. + + *Since: 2.2* + """ + ... + +**Internal method**:: + + # frobnicate(): + # + # Frobnicates this element with the specified source + # + # Args: + # source: The Source to frobnicate with + # frobilicious: Optionally specify that frobnication should be + # performed frobiliciously + # + # Returns: + # The frobnicated version of this Element. + # + def frobnicate(self, source: Source, *, frobilicious: bool = False) -> Element: + ... + +**Public API Surface instance variable**:: + + def __init__(self, context, element): + + self.name = self._compute_name(context, element) + """The name of this foo + + *Since: 2.2* + """ + +.. note:: + + Python does not support docstrings on instance variables, but sphinx does + pick them up and includes them in the generated documentation. + +**Internal instance variable**:: + + def __init__(self, context, element): + + self.name = self._compute_name(context, element) # The name of this foo + +**Internal instance variable (long)**:: + + def __init__(self, context, element): + + # This instance variable required a longer explanation, so + # it is on a line above the instance variable declaration. + self.name = self._compute_name(context, element) + + +**Public API Surface class**:: + + class Foo(Bar): + """The main Foo object in the data model + + Explanation about Foo. Note that we always document + the constructor arguments here, and not beside the __init__ + method. + + Args: + context: The invocation Context + count: The number to count + + *Since: 2.2* + """ + def __init__(self, context: Context, count: int) -> None: + ... + +**Internal class**:: + + # Foo() + # + # The main Foo object in the data model + # + # Args: + # context (Context): The invocation Context + # count (int): The number to count + # + class Foo(Bar): + ... + + +.. _contributing_class_order: + +Class structure and ordering +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When creating or modifying an object class in BuildStream, it is +important to keep in mind the order in which symbols should appear +and keep this consistent. + +Here is an example to illustrate the expected ordering of symbols +on a Python class in BuildStream:: + + class Foo(Bar): + + # Public class-wide variables come first, if any. + + # Private class-wide variables, if any + + # Now we have the dunder/magic methods, always starting + # with the __init__() method. + + def __init__(self, name): + + super().__init__() + + # NOTE: In the instance initializer we declare any instance variables, + # always declare the public instance variables (if any) before + # the private ones. + # + # It is preferred to avoid any public instance variables, and + # always expose an accessor method for it instead. + + # + # Public instance variables + # + self.name = name # The name of this foo + + # + # Private instance variables + # + self._count = 0 # The count of this foo + + ################################################ + # Abstract Methods # + ################################################ + + # NOTE: Abstract methods in BuildStream are allowed to have + # default methods. + # + # Subclasses must NEVER override any method which was + # not advertized as an abstract method by the parent class. + + # frob() + # + # Implementors should implement this to frob this foo + # count times if possible. + # + # Args: + # count (int): The number of times to frob this foo + # + # Returns: + # (int): The number of times this foo was frobbed. + # + # Raises: + # (FooError): Implementors are expected to raise this error + # + def frob(self, count): + + # + # An abstract method in BuildStream is allowed to have + # a default implementation. + # + self._count = self._do_frobbing(count) + + return self._count + + ################################################ + # Implementation of abstract methods # + ################################################ + + # NOTE: Implementations of abstract methods defined by + # the parent class should NEVER document the API + # here redundantly. + + def frobbish(self): + # + # Implementation of the "frobbish" abstract method + # defined by the parent Bar class. + # + return True + + ################################################ + # Public Methods # + ################################################ + + # NOTE: Public methods here are the ones which are expected + # to be called from outside of this class. + # + # These, along with any abstract methods, usually + # constitute the API surface of this class. + + # frobnicate() + # + # Perform the frobnication process on this Foo + # + # Raises: + # (FrobError): In the case that a frobnication error was + # encountered + # + def frobnicate(self): + frobnicator.frobnicate(self) + + # set_count() + # + # Sets the count of this foo + # + # Args: + # count (int): The new count to set + # + def set_count(self, count): + + self._count = count + + # get_count() + # + # Accessor for the count value of this foo. + # + # Returns: + # (int): The count of this foo + # + def get_count(self, count): + + return self._count + + ################################################ + # Private Methods # + ################################################ + + # NOTE: Private methods are the ones which are internal + # implementation details of this class. + # + # Even though these are private implementation + # details, they still MUST have API documenting + # comments on them. + + # _do_frobbing() + # + # Does the actual frobbing + # + # Args: + # count (int): The number of times to frob this foo + # + # Returns: + # (int): The number of times this foo was frobbed. + # + def self._do_frobbing(self, count): + return count + + +.. _contributing_public_and_private: + +Public and private symbols +~~~~~~~~~~~~~~~~~~~~~~~~~~ +BuildStream mostly follows the PEP-8 for defining *public* and *private* symbols +for any given class, with some deviations. Please read the `section on inheritance +`_ for +reference on how the PEP-8 defines public and non-public. + +* A *public* symbol is any symbol which you expect to be used by clients + of your class or module within BuildStream. + + Public symbols are written without any leading underscores. + +* A *private* symbol is any symbol which is entirely internal to your class + or module within BuildStream. These symbols cannot ever be accessed by + external clients or modules. + + A private symbol must be denoted by a leading underscore. + +* When a class can have subclasses, then private symbols should be denoted + by two leading underscores. For example, the ``Sandbox`` or ``Platform`` + classes which have various implementations, or the ``Element`` and ``Source`` + classes which plugins derive from. + + The double leading underscore naming convention invokes Python's name + mangling algorithm which helps prevent namespace collisions in the case + that subclasses might have a private symbol with the same name. + +In BuildStream, we have what we call a *"Public API Surface"*, as previously +mentioned in :ref:`contributing_documenting_symbols`. In the :ref:`next section +` we will discuss the *"Public API Surface"* and +outline the exceptions to the rules discussed here. + + +.. _contributing_public_api_surface: + +Public API surface +~~~~~~~~~~~~~~~~~~ +BuildStream exposes what we call a *"Public API Surface"* which is stable +and unchanging. This is for the sake of stability of the interfaces which +plugins use, so it can also be referred to as the *"Plugin facing API"*. + +Any symbols which are a part of the *"Public API Surface*" are never allowed +to change once they have landed in a stable release version of BuildStream. As +such, we aim to keep the *"Public API Surface"* as small as possible at all +times, and never expose any internal details to plugins inadvertently. + +One problem which arises from this is that we end up having symbols +which are *public* according to the :ref:`rules discussed in the previous section +`, but must be hidden away from the +*"Public API Surface"*. For example, BuildStream internal classes need +to invoke methods on the ``Element`` and ``Source`` classes, whereas these +methods need to be hidden from the *"Public API Surface"*. + +This is where BuildStream deviates from the PEP-8 standard for public +and private symbol naming. + +In order to disambiguate between: + +* Symbols which are publicly accessible details of the ``Element`` class, can + be accessed by BuildStream internals, but must remain hidden from the + *"Public API Surface"*. + +* Symbols which are private to the ``Element`` class, and cannot be accessed + from outside of the ``Element`` class at all. + +We denote the former category of symbols with only a single underscore, and the latter +category of symbols with a double underscore. We often refer to this distinction +as *"API Private"* (the former category) and *"Local Private"* (the latter category). + +Classes which are a part of the *"Public API Surface"* and require this disambiguation +were not discussed in :ref:`the class ordering section `, for +these classes, the *"API Private"* symbols always come **before** the *"Local Private"* +symbols in the class declaration. + +Modules which are not a part of the *"Public API Surface"* have their Python files +prefixed with a single underscore, and are not imported in BuildStream's the master +``__init__.py`` which is used by plugins. + +.. note:: + + The ``utils.py`` module is public and exposes a handful of utility functions, + however many of the functions it provides are *"API Private"*. + + In this case, the *"API Private"* functions are prefixed with a single underscore. + +Any objects which are a part of the *"Public API Surface"* should be exposed via the +toplevel ``__init__.py`` of the ``buildstream`` package. + + +File naming convention +~~~~~~~~~~~~~~~~~~~~~~ +With the exception of a few helper objects and data structures, we structure +the code in BuildStream such that every filename is named after the object it +implements. E.g. The ``Project`` object is implemented in ``_project.py``, the +``Context`` object in ``_context.py``, the base ``Element`` class in ``element.py``, +etc. + +As mentioned in the previous section, objects which are not a part of the +:ref:`public, plugin facing API surface ` have their +filenames prefixed with a leading underscore (like ``_context.py`` and ``_project.py`` +in the examples above). + +When an object name has multiple words in it, e.g. ``ArtifactCache``, then the +resulting file is named all in lower case without any underscore to separate +words. In the case of ``ArtifactCache``, the filename implementing this object +is found at ``_artifactcache/artifactcache.py``. + + +Imports +~~~~~~~ +Module imports inside BuildStream are done with relative ``.`` notation: + +**Good**:: + + from ._context import Context + +**Bad**:: + + from buildstream._context import Context + +The exception to the above rule is when authoring plugins, +plugins do not reside in the same namespace so they must +address buildstream in the imports. + +An element plugin will derive from Element by importing:: + + from buildstream import Element + +When importing utilities specifically, don't import function names +from there, instead import the module itself:: + + from . import utils + +This makes things clear when reading code that said functions +are not defined in the same file but come from utils.py for example. + + +.. _contributing_instance_variables: + +Instance variables +~~~~~~~~~~~~~~~~~~ +It is preferred that all instance state variables be declared as :ref:`private symbols +`, however in some cases, especially when the state +is immutable for the object's life time (like an ``Element`` name for example), it +is acceptable to save some typing by using a publicly accessible instance variable. + +It is never acceptable to modify the value of an instance variable from outside +of the declaring class, even if the variable is *public*. In other words, the class +which exposes an instance variable is the only one in control of the value of this +variable. + +* If an instance variable is public and must be modified; then it must be + modified using a :ref:`mutator `. + +* Ideally for better encapsulation, all object state is declared as + :ref:`private instance variables ` and can + only be accessed by external classes via public :ref:`accessors and mutators + `. + +.. note:: + + In some cases, we may use small data structures declared as objects for the sake + of better readability, where the object class itself has no real supporting code. + + In these exceptions, it can be acceptable to modify the instance variables + of these objects directly, unless they are otherwise documented to be immutable. + + +.. _contributing_accessor_mutator: + +Accessors and mutators +~~~~~~~~~~~~~~~~~~~~~~ +An accessor and mutator, are methods defined on the object class to access (get) +or mutate (set) a value owned by the declaring class, respectively. + +An accessor might derive the returned value from one or more of its components, +and a mutator might have side effects, or delegate the mutation to a component. + +Accessors and mutators are always :ref:`public ` +(even if they might have a single leading underscore and are considered +:ref:`API Private `), as their purpose is to +enforce encapsulation with regards to any accesses to the state which is owned +by the declaring class. + +Accessors and mutators are functions prefixed with ``get_`` and ``set_`` +respectively, e.g.:: + + class Foo(): + + def __init__(self): + + # Declare some internal state + self._count = 0 + + # get_count() + # + # Gets the count of this Foo. + # + # Returns: + # (int): The current count of this Foo + # + def get_foo(self): + return self._count + + # set_count() + # + # Sets the count of this Foo. + # + # Args: + # count (int): The new count for this Foo + # + def set_foo(self, count): + self._count = count + +.. attention:: + + We are aware that Python offers a facility for accessors and + mutators using the ``@property`` decorator instead. Do not use + the ``@property`` decorator. + + The decision to use explicitly defined functions instead of the + ``@property`` decorator is rather arbitrary, there is not much + technical merit to preferring one technique over the other. + However as :ref:`discussed below `, + it is of the utmost importance that we do not mix both techniques + in the same codebase. + + +.. _contributing_abstract_methods: + +Abstract methods +~~~~~~~~~~~~~~~~ +In BuildStream, an *"Abstract Method"* is a bit of a misnomer and does +not match up to how Python defines abstract methods, we need to seek out +a new nomenclature to refer to these methods. + +In Python, an *"Abstract Method"* is a method which **must** be +implemented by a subclass, whereas all methods in Python can be +overridden. + +In BuildStream, we use the term *"Abstract Method"*, to refer to +a method which **can** be overridden by a subclass, whereas it +is **illegal** to override any other method. + +* Abstract methods are allowed to have default implementations. + +* Subclasses are not allowed to redefine the calling signature + of an abstract method, or redefine the API contract in any way. + +* Subclasses are not allowed to override any other methods. + +The key here is that in BuildStream, we consider it unacceptable +that a subclass overrides a method of its parent class unless +the said parent class has explicitly given permission to subclasses +to do so, and outlined the API contract for this purpose. No surprises +are allowed. + + +Error handling +~~~~~~~~~~~~~~ +In BuildStream, all non recoverable errors are expressed via +subclasses of the ``BstError`` exception. + +This exception is handled deep in the core in a few places, and +it is rarely necessary to handle a ``BstError``. + + +Raising exceptions +'''''''''''''''''' +When writing code in the BuildStream core, ensure that all system +calls and third party library calls are wrapped in a ``try:`` block, +and raise a descriptive ``BstError`` of the appropriate class explaining +what exactly failed. + +Ensure that the original system call error is formatted into your new +exception, and that you use the Python ``from`` semantic to retain the +original call trace, example:: + + try: + os.utime(self._refpath(ref)) + except FileNotFoundError as e: + raise ArtifactError("Attempt to access unavailable artifact: {}".format(e)) from e + + +Enhancing exceptions +'''''''''''''''''''' +Sometimes the ``BstError`` originates from a lower level component, +and the code segment which raised the exception did not have enough context +to create a complete, informative summary of the error for the user. + +In these cases it is necessary to handle the error and raise a new +one, e.g.:: + + try: + extracted_artifact = self._artifacts.extract(self, cache_key) + except ArtifactError as e: + raise ElementError("Failed to extract {} while checking out {}: {}" + .format(cache_key, self.name, e)) from e + + +Programming errors +'''''''''''''''''' +Sometimes you are writing code and have detected an unexpected condition, +or a broken invariant for which the code cannot be prepared to handle +gracefully. + +In these cases, do **not** raise any of the ``BstError`` class exceptions. + +Instead, use the ``assert`` statement, e.g.:: + + assert utils._is_main_process(), \ + "Attempted to save workspace configuration from child process" + +This will result in a ``BUG`` message with the stack trace included being +logged and reported in the frontend. + + +BstError parameters +''''''''''''''''''' +When raising ``BstError`` class exceptions, there are some common properties +which can be useful to know about: + +* **message:** The brief human readable error, will be formatted on one line in the frontend. + +* **detail:** An optional detailed human readable message to accompany the **message** summary + of the error. This is often used to recommend the user some course of action, or to provide + additional context about the error. + +* **temporary:** Some errors are allowed to be *temporary*, this attribute is only + observed from child processes which fail in a temporary way. This distinction + is used to determine whether the task should be *retried* or not. An error is usually + only a *temporary* error if the cause of the error was a network timeout. + +* **reason:** A machine readable identifier for the error. This is used for the purpose + of regression testing, such that we check that BuildStream has errored out for the + expected reason in a given failure mode. + + +Documenting Exceptions +'''''''''''''''''''''' +We have already seen :ref:`some examples ` of how +exceptions are documented in API documenting comments, but this is worth some +additional disambiguation. + +* Only document the exceptions which are raised directly by the function in question. + It is otherwise nearly impossible to keep track of what exceptions *might* be raised + indirectly by calling the given function. + +* For a regular public or private method, your audience is a caller of the function; + document the exception in terms of what exception might be raised as a result of + calling this method. + +* For an :ref:`abstract method `, your audience is the + implementor of the method in a subclass; document the exception in terms of what + exception is prescribed for the implementing class to raise. + + +.. _contributing_always_consistent: + +Always be consistent +~~~~~~~~~~~~~~~~~~~~ +There are various ways to define functions and classes in Python, +which has evolved with various features over time. + +In BuildStream, we may not have leveraged all of the nice features +we could have, that is okay, and where it does not break API, we +can consider changing it. + +Even if you know there is a *better* way to do a given thing in +Python when compared to the way we do it in BuildStream, *do not do it*. + +Consistency of how we do things in the codebase is more important +than the actual way in which things are done, always. + +Instead, if you like a certain Python feature and think the BuildStream +codebase should use it, then propose your change on the `mailing list +`_. Chances +are that we will reach agreement to use your preferred approach, and +in that case, it will be important to apply the change unilaterally +across the entire codebase, such that we continue to have a consistent +codebase. + + +Avoid tail calling +~~~~~~~~~~~~~~~~~~ +With the exception of tail calling with simple functions from +the standard Python library, such as splitting and joining lines +of text and encoding/decoding text; always avoid tail calling. + +**Good**:: + + # Variables that we will need declared up top + context = self._get_context() + workspaces = context.get_workspaces() + + ... + + # Saving the workspace configuration + workspaces.save_config() + +**Bad**:: + + # Saving the workspace configuration + self._get_context().get_workspaces().save_config() + +**Acceptable**:: + + # Decode the raw text loaded from a log file for display, + # join them into a single utf-8 string and strip away any + # trailing whitespace. + return '\n'.join([line.decode('utf-8') for line in lines]).rstrip() + +When you need to obtain a delegate object via an accessor function, +either do it at the beginning of the function, or at the beginning +of a code block within the function that will use that object. + +There are several reasons for this convention: + +* When observing a stack trace, it is always faster and easier to + determine what went wrong when all statements are on separate lines. + +* We always want individual lines to trace back to their origin as + much as possible for the purpose of tracing the history of code + with *git blame*. + + One day, you might need the ``Context`` or ``Workspaces`` object + in the same function for another reason, at which point it will + be unacceptable to leave the existing line as written, because it + will introduce a redundant accessor to the same object, so the + line written as:: + + self._get_context().get_workspaces().save_config() + + Will have to change at that point, meaning we lose the valuable + information of which commit originally introduced this call + when running *git blame*. + +* For similar reasons, we prefer delegate objects be accessed near + the beginning of a function or code block so that there is less + chance that this statement will have to move in the future, if + the same function or code block needs the delegate object for any + other reason. + + Asides from this, code is generally more legible and uniform when + variables are declared at the beginning of function blocks. + + +Vertical stacking of modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +For the sake of overall comprehensiveness of the BuildStream +architecture, it is important that we retain vertical stacking +order of the dependencies and knowledge of modules as much as +possible, and avoid any cyclic relationships in modules. + +For instance, the ``Source`` objects are owned by ``Element`` +objects in the BuildStream data model, and as such the ``Element`` +will delegate some activities to the ``Source`` objects in its +possession. The ``Source`` objects should however never call functions +on the ``Element`` object, nor should the ``Source`` object itself +have any understanding of what an ``Element`` is. + +If you are implementing a low level utility layer, for example +as a part of the ``YAML`` loading code layers, it can be tempting +to derive context from the higher levels of the codebase which use +these low level utilities, instead of defining properly stand alone +APIs for these utilities to work: Never do this. + +Unfortunately, unlike other languages where include files play +a big part in ensuring that it is difficult to make a mess; Python, +allows you to just call methods on arbitrary objects passed through +a function call without having to import the module which defines +those methods - this leads to cyclic dependencies of modules quickly +if the developer does not take special care of ensuring this does not +happen. + + +Minimize arguments in methods +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When creating an object, or adding a new API method to an existing +object, always strive to keep as much context as possible on the +object itself rather than expecting callers of the methods to provide +everything the method needs every time. + +If the value or object that is needed in a function call is a constant +for the lifetime of the object which exposes the given method, then +that value or object should be passed in the constructor instead of +via a method call. + + +Minimize API surfaces +~~~~~~~~~~~~~~~~~~~~~ +When creating an object, or adding new functionality in any way, +try to keep the number of :ref:`public, outward facing ` +symbols to a minimum, this is important for both +:ref:`internal and public, plugin facing API surfaces `. + +When anyone visits a file, there are two levels of comprehension: + +* What do I need to know in order to *use* this object. + +* What do I need to know in order to *modify* this object. + +For the former, we want the reader to understand with as little effort +as possible, what the public API contract is for a given object and consequently, +how it is expected to be used. This is also why we +:ref:`order the symbols of a class ` in such a way +as to keep all outward facing public API surfaces at the top of the file, so that the +reader never needs to dig deep into the bottom of the file to find something they +might need to use. + +For the latter, when it comes to having to modify the file or add functionality, +you want to retain as much freedom as possible to modify internals, while +being sure that nothing external will be affected by internal modifications. +Less client facing API means that you have less surrounding code to modify +when your API changes. Further, ensuring that there is minimal outward facing +API for any module minimizes the complexity for the developer working on +that module, by limiting the considerations needed regarding external side +effects of their modifications to the module. + +When modifying a file, one should not have to understand or think too +much about external side effects, when the API surface of the file is +well documented and minimal. + +When adding new API to a given object for a new purpose, consider whether +the new API is in any way redundant with other API (should this value now +go into the constructor, since we use it more than once? could this +value be passed along with another function, and the other function renamed, +to better suit the new purposes of this module/object?) and repurpose +the outward facing API of an object as a whole every time. + + +Avoid transient state on instances +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +At times, it can be tempting to store transient state that is +the result of one operation on an instance, only to be retrieved +later via an accessor function elsewhere. + +As a basic rule of thumb, if the value is transient and just the +result of one operation, which needs to be observed directly after +by another code segment, then never store it on the instance. + +BuildStream is complicated in the sense that it is multi processed +and it is not always obvious how to pass the transient state around +as a return value or a function parameter. Do not fall prey to this +obstacle and pollute object instances with transient state. + +Instead, always refactor the surrounding code so that the value +is propagated to the desired end point via a well defined API, either +by adding new code paths or changing the design such that the +architecture continues to make sense. + + +Refactor the codebase as needed +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Especially when implementing features, always move the BuildStream +codebase forward as a whole. + +Taking a short cut is alright when prototyping, but circumventing +existing architecture and design to get a feature implemented without +re-designing the surrounding architecture to accommodate the new +feature instead, is never acceptable upstream. + +For example, let's say that you have to implement a feature and you've +successfully prototyped it, but it launches a ``Job`` directly from a +``Queue`` implementation to get the feature to work, while the ``Scheduler`` +is normally responsible for dispatching ``Jobs`` for the elements on +a ``Queue``. This means that you've proven that your feature can work, +and now it is time to start working on a patch for upstream. + +Consider what the scenario is and why you are circumventing the design, +and then redesign the ``Scheduler`` and ``Queue`` objects to accommodate for +the new feature and condition under which you need to dispatch a ``Job``, +or how you can give the ``Queue`` implementation the additional context it +needs. diff --git a/_sources/hacking/grpc_protocols.rst.txt b/_sources/hacking/grpc_protocols.rst.txt new file mode 100644 index 000000000..0ca4c743b --- /dev/null +++ b/_sources/hacking/grpc_protocols.rst.txt @@ -0,0 +1,38 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _protocol_buffers: + +Generating protocol buffers +--------------------------- +BuildStream uses protobuf and gRPC for serialization and communication with +artifact cache servers. This requires ``.proto`` files and Python code +generated from the ``.proto`` files using protoc. All these files live in the +``src/buildstream/_protos`` directory. The generated files are included in the +git repository to avoid depending on grpcio-tools for user installations. + + +Regenerating code +~~~~~~~~~~~~~~~~~ +When ``.proto`` files are modified, the corresponding Python code needs to +be regenerated. As a prerequisite for code generation you need to install +``grpcio-tools`` using pip or some other mechanism:: + + pip3 install --user grpcio-tools + +To actually regenerate the code:: + + ./setup.py build_grpc + diff --git a/_sources/hacking/making_releases.rst.txt b/_sources/hacking/making_releases.rst.txt new file mode 100644 index 000000000..4f71c4521 --- /dev/null +++ b/_sources/hacking/making_releases.rst.txt @@ -0,0 +1,266 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _making_releases: + +Making releases +--------------- +This is a checklist of activities which must be observed when creating +BuildStream releases, it is important to keep this section up to date +whenever the release process changes. + + +Requirements +~~~~~~~~~~~~ +There are a couple of requirements and accounts required in order +to publish a release. + +* Ability to send email to ``dev@buildstream.apache.org``. + +* Shell account at ``master.gnome.org``. + +* Access to the `BuildStream project on PyPI `_ + +* An email client which still knows how to send emails in plain text. + + +Pre-release changes +~~~~~~~~~~~~~~~~~~~ +Before actually rolling the release, here is a list of changes which +might need to be done in preparation of the release. + +* Ensure that the man pages are up to date + + The man pages are committed to the repository because we are + currently unable to integrate this generation into the setuptools + build phase, as outlined in issue #8. + + If any of the user facing CLI has changed, or if any of the + related docstrings have changed, then you should + :ref:`regenerate the man pages ` and + add/commit the results before wrapping a release. + +* Ensure the documentation session HTML is up to date + + The session HTML files are committed to the repository for multiple + reasons, one of them being that the documentation must be buildable + from within a release build environment so that downstream distribution + packagers can easily create the docs package. + + This is currently only needed for the first stable release + in a stable line of releases, after this point the API is frozen + and will not change for the remainder of the stable release lifetime, + so nothing interesting will have changed in these session files. + + If regeneration is needed, follow :ref:`the instructions above `. + +* Ensure the NEWS entry is up to date and ready + + For a stable release where features have not been added, we + should at least add some entries about the issues which have + been fixed since the last stable release. + + For development releases, it is worthwhile going over the + existing entries and ensuring all the major feature additions + are mentioned and there are no redundancies. + +* Push pre-release changes + + Now that any final pre-release changes to generated files or NEWS have + been made, push these directly to the upstream repository. + + Do not sit around waiting for CI or approval, these superficial changes + do not affect CI and you are intended to push these changes directly + to the upstream repository. + + +Release process +~~~~~~~~~~~~~~~ + +* Ensure that the latest commit is passing in CI + + Of course, we do not release software which does not pass it's own + tests. + +* Get the list of contributors + + The list of contributors for a given list is a list of + any contributors who have landed any patches since the + last release. + + An easy way to get this list is to ask git to summarize + the authors of commits since the *last release tag*. For + example, if we are about to create the ``1.1.1`` release, then + we need to observe all of the commits since the ``1.1.0`` + release: + + .. code:: shell + + git shortlog -s 1.1.0...@ + + At times, the same contributor might make contributions from different + machines which they have setup their author names differently, you + can see that some of the authors are actually duplicates, then + remove the duplicates. + +* Start composing the release announcement email + + The first thing to do when composing the release email is to + ensure your mail client has disabled any HTML formatting and will + safely use plain text only. + + Try to make the release announcement consistent with other release + announcements as much as possible, an example of the email + can be `found here `_. + + The recipient of the email is ``dev@buildstream.apache.org`` and the title + of the email should be of the form: ``BuildStream 1.1.1 released``, without + any exclamation point. + + The format of the email is essentially:: + + Hi all, + + This is the personalized message written to you about this + release. + + If this is an unstable release, this should include a warning + to this effect and an invitation to users to please help us + test this release. + + This is also a good place to highlight specific bug fixes which + users may have been waiting for, or highlight a new feature we + want users to try out. + + + What is BuildStream ? + ===================== + This is a concise blurb which describes BuildStream in a couple of + sentences, and is taken from the the README.rst. + + The easiest thing is to just copy this over from the last release email. + + + ================= + buildstream 1.1.1 + ================= + This section is directly copy pasted from the top of the NEWS file + + + Contributors + ============ + - This is Where + - You Put + - The Contributor + - Names Which + - You Extracted + - Using git shortlog -s + + + Where can I get it ? + ==================== + https://download.gnome.org/sources/BuildStream/1.1/ + + For more information on the BuildStream project, visit our home page + at https://buildstream.build/ + +* Publish the release tag + + Now that any pre-release changes are upstream, create and push the + signed release tag like so: + + .. code:: shell + + git tag -s 1.1.1 + git push origin 1.1.1 + + This will trigger the "Release actions" workflow which also takes care of: + + * uploading Github release artifacts + * uploading Python source and binary packages to PyPI + +* Upload the release tarball + + First get yourself into a clean repository state, ensure that you + don't have any unfinished work or precious, uncommitted files lying + around in your checkout and then run: + + .. code:: shell + + git clean -xdff + + Create the tarball with the following command: + + .. code:: shell + + python3 setup.py sdist + + And upload the resulting tarball to the master GNOME server: + + .. code:: shell + + scp dist/BuildStream-1.1.1.tar.gz @master.gnome.org: + + And finally login to your account at ``master.gnome.org`` and run + the install scripts to publish the tarball and update the mirrors: + + .. code:: shell + + ftpadmin install BuildStream-1.1.1.tar.gz + +* Send the release email + + Now that the release tag is up and the tarball is published, + you can send the release email. + + +Post-release activities +~~~~~~~~~~~~~~~~~~~~~~~ +Once the release has been published, there are some activities +which need to be done to ensure everything is up to date. + +* Check that the release was successfully uploaded to PyPI. If + it is an unstable release, make sure the version on PyPI has + the correct "dev0" postfix so to avoid being treated as stable. + +* Update the topic line in the #buildstream IRC channel if needed + + The IRC channel usually advertizes the latest stable release + in the topic line, now is the right time to update it. + +* Update the website repository + + The website wants to link to release announcements, but this + cannot be automated because we cannot guess what the link to + the release email will be in the mailing list archive. + + Find the URL to the announcement you just published + `in the mailing list archives `_, + and use that URL to update the ``anouncements.json`` file in the website + repository. + + Commit and push this change to the the ``anouncements.json`` file to + the upstream website repository, and gitlab will take care of automatically + updating the website accordingly. + +* Regenerate BuildStream documentation + + In order to update the badges we use in various documentation + which reflects what is the latest stable releases and the latest + development snapshots, we simply need to ensure a pipeline runs + for the master branch in the BuildStream repository. + + You can do this by using the "Run Pipeline" feature on the + `pipelines page in the gitlab UI `_. diff --git a/_sources/hacking/managing_data_files.rst.txt b/_sources/hacking/managing_data_files.rst.txt new file mode 100644 index 000000000..291ee3913 --- /dev/null +++ b/_sources/hacking/managing_data_files.rst.txt @@ -0,0 +1,28 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _managing_data_files: + +Managing data files +------------------- +When adding data files which need to be discovered at runtime by BuildStream, update setup.py accordingly. + +When adding data files for the purpose of docs or tests, or anything that is not covered by +setup.py, update the MANIFEST.in accordingly. + +At any time, running the following command to create a source distribution should result in +creating a tarball which contains everything we want it to include:: + + ./setup.py sdist diff --git a/_sources/hacking/measuring_performance.rst.txt b/_sources/hacking/measuring_performance.rst.txt new file mode 100644 index 000000000..44798c4d4 --- /dev/null +++ b/_sources/hacking/measuring_performance.rst.txt @@ -0,0 +1,108 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _measuring_performance: + +Measuring performance +--------------------- + +Benchmarking framework +~~~~~~~~~~~~~~~~~~~~~~~ +BuildStream has a utility to measure performance which is available from a +separate repository at https://gitlab.com/BuildStream/benchmarks. This tool +allows you to run a fixed set of workloads with multiple versions of +BuildStream. From this you can see whether one version performs better or +worse than another which is useful when looking for regressions and when +testing potential optimizations. + +For full documentation on how to use the benchmarking tool see the README in +the 'benchmarks' repository. + + +Profiling tools +~~~~~~~~~~~~~~~ +When looking for ways to speed up the code you should make use of a profiling +tool. + +Python provides `cProfile `_ +which gives you a list of all functions called during execution and how much +time was spent in each function. Here is an example of running ``bst --help`` +under cProfile: + + python3 -m cProfile -o bst.cprofile -- $(which bst) --help + +You can then analyze the results interactively using the 'pstats' module: + + python3 -m pstats ./bst.cprofile + +For more detailed documentation of cProfile and 'pstats', see: +https://docs.python.org/3/library/profile.html. + +For a richer and interactive visualisation of the `.cprofile` files, you can +try `snakeviz `_. +You can install it with `pip install snakeviz`. Here is an example invocation: + + snakeviz bst.cprofile + +It will then start a webserver and launch a browser to the relevant page. + +.. note:: + + If you want to also profile cython files, you will need to add + BST_CYTHON_PROFILE=1 and recompile the cython files. + ``pip install`` would take care of that. + +Profiling specific parts of BuildStream with BST_PROFILE +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +BuildStream can also turn on cProfile for specific parts of execution +using BST_PROFILE. + +BST_PROFILE can be set to a section name, or a list of section names separated +by ":". You can also use "all" for getting all profiles at the same time. +There is a list of topics in `src/buildstream/_profile.py`. For example, running:: + + BST_PROFILE=load-pipeline bst build bootstrap-system-x86.bst + +will produce a profile in the current directory for the time take to +call most of `initialized`, for each element. These profile files +are in the same cProfile format as those mentioned in the previous +section, and can be analysed in the same way. + +Fixing performance issues +~~~~~~~~~~~~~~~~~~~~~~~~~ + +BuildStream uses `Cython `_ in order to speed up specific parts of the +code base. + +.. note:: + + When optimizing for performance, please ensure that you optimize the algorithms before + jumping into Cython code. Cython will make the code harder to maintain and less accessible + to all developers. + +When adding a new cython file to the codebase, you will need to register it in ``setup.py``. + +For example, for a module ``buildstream._my_module``, to be written in ``src/buildstream/_my_module.pyx``, you would do:: + + register_cython_module("buildstream._my_module") + +In ``setup.py`` and the build tool will automatically use your module. + +.. note:: + + Please register cython modules at the same place as the others. + +When adding a definition class to share cython symbols between modules, please document the implementation file +and only expose the required definitions. diff --git a/_sources/hacking/ui.rst.txt b/_sources/hacking/ui.rst.txt new file mode 100644 index 000000000..5a8c4d5c9 --- /dev/null +++ b/_sources/hacking/ui.rst.txt @@ -0,0 +1,90 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +Contributing to the UI +~~~~~~~~~~~~~~~~~~~~~~ + +As we wish to cleanly separate BuildStream's core from the frontend, anything +user facing should be defined within the modules contained within the `_frontend +`_ +directory. + +BuildStream's frontend includes: + + * Implementation of the command line interface (cli.py) + * Logline/text formatting (widget.py) + * The main application state (app.py) which initializes the stream to handle + logging and user interactions. + * Colour profiles (profile.py) + * Rendering of the status bar (status.py) + * Autocompletion behaviour (completions.py) + + +The command line interface +'''''''''''''''''''''''''' +All of BuildStream's commands are defined within the module `cli.py +`_ -, +the main entry point which implements the CLI. + +The command line interface is generated with `Click +`_ - a third party Python package. Click +is easy to use and automatically generates help pages. + +When working with commands, please adhere to the following checklist: + +1. All commands should be defined with a help text +2. The command should be placed within the appropriate sub group + + - If the command manipulates sources, it should be part of the + :ref:`source_subcommands`. + - If the command manipulates cached artifacts, it should be part of the + :ref:`artifact_subcommands`. + - If the command has anything to do with workspaces, it should be part + of the :ref:`workspace_subcommands`. + +3. If the command is intended to work with artifact refs as well as element + names, the command's argument should be "artifacts" as this supports the + auto-completion of artifact refs. +4. The supported `--deps` options are: "run", "build", "all", "plan" and "none". + These are always of type `click.Choice + `_ and + should always be specified with a default. If the default is "none", note that + we use the string "none", not the Python built-in ``None``. In addition to this, + the ``show_default`` flag should be set to ``True``. +5. Commands should use the app and go through the stream (via a similarly named + method within Stream) in order to communicate to BuildStream's core. + + +Displaying information +'''''''''''''''''''''' + +Output which we wish to display to the user from the frontend should use the +implemented classes in widget.py. This module contains classes which represent +widgets for displaying information to the user. + +To report messages back to the frontend, we use the ``Message()`` object +which is available from the ``Context``. + +Supported message types are defined in `_message.py +`_ +and various uses of the messenger are defined in `_messenger.py +`_ + +The ``Messenger`` class defines various methods which allow us to report back to +the frontend in particular ways. The common methods are: + +* ``Messenger.message()`` - the central point through which all messages pass +* ``Messenger.timed_activity()`` - a context manager for performing and logging + timed activities. +* ``Messenger.simple_task()`` - a Context manager for creating a task to report + progress too. diff --git a/_sources/hacking/updating_python_deps.rst.txt b/_sources/hacking/updating_python_deps.rst.txt new file mode 100644 index 000000000..548c3175e --- /dev/null +++ b/_sources/hacking/updating_python_deps.rst.txt @@ -0,0 +1,136 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _updating_python_deps: + +Updating BuildStream's Python dependencies +------------------------------------------ +BuildStream's Python dependencies are listed in multiple +`requirements files `_ +present in the ``requirements`` directory. + +All ``.txt`` files in this directory are generated from the corresponding +``.in`` file, and each ``.in`` file represents a set of dependencies. For +example, ``requirements.in`` contains all runtime dependencies of BuildStream. +``requirements.txt`` is generated from it, and contains pinned versions of all +runtime dependencies (including transitive dependencies) of BuildStream. + +When adding a new dependency to BuildStream, or updating existing dependencies, +it is important to update the appropriate requirements file accordingly. After +changing the ``.in`` file, run the following to update the matching ``.txt`` +file:: + + make -C requirements + + +Adding support for a new Python release +--------------------------------------- +When a new stable release of Python 3 appears, we must explicitly declare +our support for it in the following places. + + +tox.ini +~~~~~~~ +The ``tox.ini`` file defines the environments where the BuildStream test suite +runs. Every ``py{3.x,3.y}`` list must be updated to contain the new version +number such as ``311`` for CPython 3.11. + +Use ``tox -e py311-nocover`` to run the test suite with the new version of +Python. + + +pyproject.toml +~~~~~~~~~~~~~~ + + +Bump cython version +''''''''''''''''''' +New releases of Cython must be depended on with new versions of Python +in lock step. + +When supporting a new Python version, it is important to bump the minimal +dependency on Cython to a new enough version which also supports the new +version of Python. + + +Wheel details +''''''''''''' +We produce binary "wheel" packages for each supported version of Python. +The cibuildwheel build tool will build for all released versions of Python +so no change is needed in the config. + +However, if you want to test wheel building with a prerelease version of Python +you will need to set ``CIBW_PRERELEASE_PYTHONS=1`` in the cibuildwheel +environment. + + +.github/compose/ci.docker-compose.yml +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each binary package is tested in a container, using the +`pypa/manylinux `_ images. + +You need to add a new docker-compose service here -- copy the +latest one and update the version number where it appears. + + +.github/workflows/ci.yml and .github/workflows/release.yml +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +There is a separate CI job to run each of the above tests. Update the +matrix config for the ``test_wheels`` jobs in ``ci.yml`` and ``release.yml`` +to add the new Python version. + + +Removing support for a Python release +------------------------------------- + + +tox.ini +~~~~~~~ +You will need to update the ``py{3.x,3.y}`` lists to remove the old version. In +the ``envlist`` section, make sure the oldest version still has coverage +enabled while the other versions are marked ``-nocover``. + + +pyproject.toml +~~~~~~~~~~~~~~ +The cibuildwheel tool will produce wheels for all versions of Python supported +upstream.. If we drop support for a version before upstream do, update the +``tool.cibuildwheel.skip`` list to skip all platform tags for that version. +The glob ``cp36-*`` would skip all CPython 3.6 builds, for example. + + +.github/compose/ci.docker-compose.yml +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Remove the corresponding service. + + +.github/workflows/ci.yml and .github/workflows/release.yml +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Update the matrix config for the `test_wheels` jobs in `ci.yml` and +`release.yml` to remove the old Python version. + + +ABI compatibility for binary Python packages +-------------------------------------------- +The Python binary packages declare system requirements using +`platform compatibility tags `_. + +For linux-gnu systems we use `manylinux_x_y platform tags `_ +to specify a minimum GLIBC version. The platform tag is controlled in ``pyproject.toml`` with the +``tool.cibuildwheel.manylinux-x86_64-image`` key. It must correspond with the version of +GLIBC used in `buildbox-integration `_ +to produce static buildbox binaries that are included in the package. +The ``cibuildwheel`` tool uses `auditwheel `_ +to ensure the correct platform tag is declared. diff --git a/_sources/hacking/using_the_testsuite.rst.txt b/_sources/hacking/using_the_testsuite.rst.txt new file mode 100644 index 000000000..347b4ac82 --- /dev/null +++ b/_sources/hacking/using_the_testsuite.rst.txt @@ -0,0 +1,282 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _contributing_testing: + +Using the test suite +-------------------- +BuildStream uses `tox `_ as a frontend to run the +tests which are implemented using `pytest `_. We use +pytest for regression tests and testing out the behavior of newly added +components. + +The elaborate documentation for pytest can be found here: http://doc.pytest.org/en/latest/contents.html + +Don't get lost in the docs if you don't need to, follow existing examples instead. + + +.. _contributing_build_deps: + +Installing build dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Some of BuildStream's dependencies have non-python build dependencies. When +running tests with ``tox``, you will first need to install these dependencies. +Exact steps to install these will depend on your operating system. Commands +for installing them for some common distributions are listed below. + +For Fedora-based systems:: + + dnf install gcc python3-devel + + +For Debian-based systems:: + + apt install gcc python3-dev + + +Installing runtime dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To be able to run BuildStream as part of the test suite, BuildStream's runtime +dependencies must also be installed. Instructions on how to do so can be found +in :ref:`install-dependencies`. + +If you are not interested in running the integration tests, you can skip the +installation of ``buildbox-run``. + + +Running tests +~~~~~~~~~~~~~ +To run the tests, simply navigate to the toplevel directory of your BuildStream +checkout and run:: + + tox + +By default, the test suite will be run against every supported python version +found on your host. If you have multiple python versions installed, you may +want to run tests against only one version and you can do that using the ``-e`` +option when running tox:: + + tox -e py37 + +If you would like to test and lint at the same time, or if you do have multiple +python versions installed and would like to test against multiple versions, then +we recommend using `detox `_, just run it with +the same arguments you would give `tox`:: + + detox -e lint,py36,py37 + +The output of all failing tests will always be printed in the summary, but +if you want to observe the stdout and stderr generated by a passing test, +you can pass the ``-s`` option to pytest as such:: + + tox -- -s + +.. tip:: + + The ``-s`` option is `a pytest option `_. + + Any options specified before the ``--`` separator are consumed by ``tox``, + and any options after the ``--`` separator will be passed along to pytest. + +You can always abort on the first failure by running:: + + tox -- -x + +Similarly, you may also be interested in the ``--last-failed`` and +``--failed-first`` options as per the +`pytest cache `_ documentation. + +If you want to run a specific test or a group of tests, you +can specify a prefix to match. E.g. if you want to run all of +the frontend tests you can do:: + + tox -- tests/frontend/ + +Specific tests can be chosen by using the :: delimiter after the test module. +If you wanted to run the test_build_track test within frontend/buildtrack.py you could do:: + + tox -- tests/frontend/buildtrack.py::test_build_track + +When running only a few tests, you may find the coverage and timing output +excessive, there are options to trim them. Note that coverage step will fail. +Here is an example:: + + tox -- --no-cov --durations=1 tests/frontend/buildtrack.py::test_build_track + +We also have a set of slow integration tests that are disabled by +default - you will notice most of them marked with SKIP in the pytest +output. To run them, you can use:: + + tox -- --integration + +In case BuildStream's dependencies were updated since you last ran the +tests, you might see some errors like +``pytest: error: unrecognized arguments: --codestyle``. If this happens, you +will need to force ``tox`` to recreate the test environment(s). To do so, you +can run ``tox`` with ``-r`` or ``--recreate`` option. + +.. note:: + + By default, we do not allow use of site packages in our ``tox`` + configuration to enable running the tests in an isolated environment. + If you need to enable use of site packages for whatever reason, you can + do so by passing the ``--sitepackages`` option to ``tox``. Also, you will + not need to install any of the build dependencies mentioned above if you + use this approach. + +.. note:: + + While using ``tox`` is practical for developers running tests in + more predictable execution environments, it is still possible to + execute the test suite against a specific installation environment + using pytest directly:: + + pytest + + If you want to run coverage, you will need need to add ``BST_CYTHON_TRACE=1`` + to your environment if you also want coverage on cython files. You could then + get coverage by running:: + + BST_CYTHON_TRACE=1 coverage run pytest + + Note that you will have to have all dependencies installed already, when + running tests directly via ``pytest``. This includes the following: + + * Cython and Setuptools, as build dependencies + * Runtime dependencies and test dependencies are specified in requirements + files, present in the ``requirements`` subdirectory. Refer to the ``.in`` + files for loose dependencies and ``.txt`` files for fixed version of all + dependencies that are known to work. + * Additionally, if you are running tests that involve external plugins, you + will need to have those installed as well. + + You can also have a look at our tox configuration in ``tox.ini`` file if you + are unsure about dependencies. + +.. tip:: + + We also have an environment called 'venv' which takes any arguments + you give it and runs them inside the same virtualenv we use for our + tests:: + + tox -e venv -- + + Any commands after ``--`` will be run a virtualenv managed by tox. + +Running linters +~~~~~~~~~~~~~~~ +Linting is performed separately from testing. In order to run the linting step which +consists of running the ``pylint`` tool, run the following:: + + tox -e lint + +.. tip:: + + The project specific pylint configuration is stored in the toplevel + buildstream directory in the ``.pylintrc`` file. This configuration can be + interesting to use with IDEs and other developer tooling. + +.. _contributing_formatting_code: + +Formatting code +~~~~~~~~~~~~~~~ +Similar to linting, code formatting is also done via a ``tox`` environment. To +format the code using the ``black`` tool, run the following:: + + tox -e format + +Observing coverage +~~~~~~~~~~~~~~~~~~ +Once you have run the tests using `tox` (or `detox`), some coverage reports will +have been left behind. + +To view the coverage report of the last test run, simply run:: + + tox -e coverage + +This will collate any reports from separate python environments that may be +under test before displaying the combined coverage. + + +Adding tests +~~~~~~~~~~~~ +Tests are found in the tests subdirectory, inside of which +there is a separate directory for each *domain* of tests. +All tests are collected as:: + + tests/*/*.py + +If the new test is not appropriate for the existing test domains, +then simply create a new directory for it under the tests subdirectory. + +Various tests may include data files to test on, there are examples +of this in the existing tests. When adding data for a test, create +a subdirectory beside your test in which to store data. + +When creating a test that needs data, use the datafiles extension +to decorate your test case (again, examples exist in the existing +tests for this), documentation on the datafiles extension can +be found here: https://pypi.python.org/pypi/pytest-datafiles. + +Tests that run a sandbox should be decorated with:: + + @pytest.mark.integration + +and use the integration cli helper. + +You must test your changes in an end-to-end fashion. Consider the first end to +be the appropriate user interface, and the other end to be the change you have +made. + +The aim for our tests is to make assertions about how you impact and define the +outward user experience. You should be able to exercise all code paths via the +user interface, just as one can test the strength of rivets by sailing dozens +of ocean liners. Keep in mind that your ocean liners could be sailing properly +*because* of a malfunctioning rivet. End-to-end testing will warn you that +fixing the rivet will sink the ships. + +The primary user interface is the cli, so that should be the first target 'end' +for testing. Most of the value of BuildStream comes from what you can achieve +with the cli. + +We also have what we call a *"Public API Surface"*, as previously mentioned in +:ref:`contributing_documenting_symbols`. You should consider this a secondary +target. This is mainly for advanced users to implement their plugins against. + +Note that both of these targets for testing are guaranteed to continue working +in the same way across versions. This means that tests written in terms of them +will be robust to large changes to the code. This important property means that +BuildStream developers can make large refactorings without needing to rewrite +fragile tests. + +Another user to consider is the BuildStream developer, therefore internal API +surfaces are also targets for testing. For example the YAML loading code, and +the CasCache. Remember that these surfaces are still just a means to the end of +providing value through the cli and the *"Public API Surface"*. + +It may be impractical to sufficiently examine some changes in an end-to-end +fashion. The number of cases to test, and the running time of each test, may be +too high. Such typically low-level things, e.g. parsers, may also be tested +with unit tests; alongside the mandatory end-to-end tests. + +It is important to write unit tests that are not fragile, i.e. in such a way +that they do not break due to changes unrelated to what they are meant to test. +For example, if the test relies on a lot of BuildStream internals, a large +refactoring will likely require the test to be rewritten. Pure functions that +only rely on the Python Standard Library are excellent candidates for unit +testing. + +Unit tests only make it easier to implement things correctly, end-to-end tests +make it easier to implement the right thing. diff --git a/_sources/hacking/writing_documentation.rst.txt b/_sources/hacking/writing_documentation.rst.txt new file mode 100644 index 000000000..f744c6311 --- /dev/null +++ b/_sources/hacking/writing_documentation.rst.txt @@ -0,0 +1,299 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _writing_documentation: + +Writing documentation +--------------------- +BuildStream starts out as a documented project from day one and uses +`sphinx `_ to document itself. + +This section discusses formatting policies for editing files in the +``doc/source`` directory, and describes the details of how the docs are +generated so that you can easily generate and view the docs yourself before +submitting patches to the documentation. + +For details on how API documenting comments and docstrings are formatted, +refer to the :ref:`documenting section of the coding guidelines +`. + + +Documentation formatting policy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The BuildStream documentation style is as follows: + +* Titles and headings require two leading empty lines above them. + Only the first word in a title should be capitalized. + + * If there is an ``.. _internal_link:`` anchor, there should be two empty lines + above the anchor, followed by one leading empty line. + +* Within a section, paragraphs should be separated by one empty line. + +* Notes are defined using: ``.. note::`` blocks, followed by an empty line + and then indented (3 spaces) text. + + * Other kinds of notes can be used throughout the documentation and will + be decorated in different ways, these work in the same way as ``.. note::`` does. + + Feel free to also use ``.. attention::`` or ``.. important::`` to call special + attention to a paragraph, ``.. tip::`` to give the reader a special tip on how + to use an advanced feature or ``.. warning::`` to warn the user about a potential + misuse of the API and explain its consequences. + +* Code blocks are defined using: ``.. code:: LANGUAGE`` blocks, followed by an empty + line and then indented (3 spaces) text. Note that the default language is ``python``. + +* Cross references should be of the form ``:role:`target```. + + * Explicit anchors can be declared as ``.. _anchor_name:`` on a line by itself. + + * To cross reference arbitrary locations with, for example, the anchor ``anchor_name``, + always provide some explicit text in the link instead of deriving the text from + the target, e.g.: ``:ref:`Link text ```. + Note that the "_" prefix is not used when referring to the target. + +For further information about using the reStructuredText with sphinx, please see the +`Sphinx Documentation `_. + + +Building Docs +~~~~~~~~~~~~~ +Before you can build the docs, you will end to ensure that you have installed +the required :ref:`build dependencies ` as mentioned +in the testing section above. + +To build the documentation, just run the following:: + + tox -e docs + +This will give you a ``doc/build/html`` directory with the html docs which +you can view in your browser locally to test. + + +.. _contributing_session_html: + +Regenerating session html +''''''''''''''''''''''''' +The documentation build will build the session files if they are missing, +or if explicitly asked to rebuild. We revision the generated session html files +in order to reduce the burden on documentation contributors. + +To explicitly rebuild the session snapshot html files, it is recommended that you +first set the ``BST_SOURCE_CACHE`` environment variable to your source cache, this +will make the docs build reuse already downloaded sources:: + + export BST_SOURCE_CACHE=~/.cache/buildstream/sources + +To force rebuild session html while building the doc, simply run `tox` with the +``BST_FORCE_SESSION_REBUILD`` environment variable set, like so:: + + env BST_FORCE_SESSION_REBUILD=1 tox -e docs + + +.. _contributing_man_pages: + +Man pages +~~~~~~~~~ +Unfortunately it is quite difficult to integrate the man pages build +into the ``setup.py``, as such, whenever the frontend command line +interface changes, the static man pages should be regenerated and +committed with that. + +To do this, run the following from the the toplevel directory of BuildStream:: + + tox -e man + +And commit the result, ensuring that you have added anything in +the ``man/`` subdirectory, which will be automatically included +in the buildstream distribution. + + +User guide +~~~~~~~~~~ +The :ref:`user guide ` is comprised of free form documentation +in manually written ``.rst`` files and is split up into a few sections, +of main interest are the :ref:`tutorial ` and the +:ref:`examples `. + +The distinction of the two categories of user guides is important to +understand too. + +* **Tutorial** + + The tutorial is structured as a series of exercises which start with + the most basic concepts and build upon the previous chapters in order + to arrive at a basic understanding of how to create BuildStream projects. + + This series of examples should be easy enough to complete in a matter + of a few hours for a new user, and should provide just enough insight to + get the user started in creating their own projects. + + Going through the tutorial step by step should also result in the user + becoming proficient enough with the reference manual to get by on their own. + +* **Examples** + + These exist to demonstrate how to accomplish more advanced tasks which + are not always obvious and discoverable. + + Alternatively, these also demonstrate elegant and recommended ways of + accomplishing some tasks which could be done in various ways. + + +Guidelines +'''''''''' +Here are some general guidelines for adding new free form documentation +to the user guide. + +* **Focus on a single subject** + + It is important to stay focused on a single subject and avoid getting + into tangential material when creating a new entry, so that the articles + remain concise and the user is not distracted by unrelated subject material. + + A single tutorial chapter or example should not introduce any additional + subject material than the material being added for the given example. + +* **Reuse existing sample project elements** + + To help avoid distracting from the topic at hand, it is always preferable to + reuse the same project sample material from other examples and only deviate + slightly to demonstrate the new material, than to create completely new projects. + + This helps us remain focused on a single topic at a time, and reduces the amount + of unrelated material the reader needs to learn in order to digest the new + example. + +* **Don't be redundant** + + When something has already been explained in the tutorial or in another example, + it is best to simply refer to the other user guide entry in a new example. + + Always prefer to link to the tutorial if an explanation exists in the tutorial, + rather than linking to another example, where possible. + +* **Link into the reference manual at every opportunity** + + The format and plugin API is 100% documented at all times. Whenever discussing + anything about the format or plugin API, always do so while providing a link + into the more terse reference material. + + We don't want users to have to search for the material themselves, and we also + want the user to become proficient at navigating the reference material over + time. + +* **Use concise terminology** + + As developers, we tend to come up with code names for features we develop, and + then end up documenting a new feature in an example. + + Never use a code name or shorthand to refer to a feature in the user guide, instead + always use fully qualified sentences outlining very explicitly what we are doing + in the example, or what the example is for in the case of a title. + + We need to be considerate that the audience of our user guide is probably a + proficient developer or integrator, but has no idea what we might have decided + to name a given activity. + + +Structure of an example +''''''''''''''''''''''' +The :ref:`tutorial ` and the :ref:`examples ` sections +of the documentation contain a series of sample projects, each chapter in +the tutorial, or standalone example uses a sample project. + +Here is the the structure for adding new examples and tutorial chapters. + +* The example has a ``${name}``. + +* The example has a project users can copy and use. + + * This project is added in the directory ``doc/examples/${name}``. + +* The example has a documentation component. + + * This is added at ``doc/source/examples/${name}.rst`` + * An entry for ``examples/${name}`` is added to the toctree in ``doc/source/using_examples.rst`` + * This documentation discusses the project elements declared in the project and may + provide some BuildStream command examples. + * This documentation links out to the reference manual at every opportunity. + + .. note:: + + In the case of a tutorial chapter, the ``.rst`` file is added in at + ``doc/source/tutorial/${name}.rst`` and an entry for ``tutorial/${name}`` + is added to ``doc/source/using_tutorial.rst``. + +* The example has a CI test component. + + * This is an integration test added at ``tests/examples/${name}``. + * This test runs BuildStream in the ways described in the example + and assert that we get the results which we advertize to users in + the said examples. + + +Adding BuildStream command output +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +As a part of building the docs, BuildStream will run itself and extract +some html for the colorized output which is produced. + +If you want to run BuildStream to produce some nice html for your +documentation, then you can do so by adding new ``.run`` files to the +``doc/sessions/`` directory. + +Any files added as ``doc/sessions/${example}.run`` will result in generated +file at ``doc/source/sessions/${example}.html``, and these files can be +included in the reStructuredText documentation at any time with:: + + .. raw:: html + :file: sessions/${example}.html + +The ``.run`` file format is just another YAML dictionary which consists of a +``commands`` list, instructing the program what to do command by command. + +Each *command* is a dictionary, the members of which are listed here: + +* ``directory``: The input file relative project directory. + +* ``output``: The input file relative output html file to generate (optional). + +* ``fake-output``: Don't really run the command, just pretend to and pretend + this was the output, an empty string will enable this too. + +* ``command``: The command to run, without the leading ``bst``. + +* ``shell``: Specifying ``True`` indicates that ``command`` should be run as + a shell command from the project directory, instead of a bst command (optional). + +When adding a new ``.run`` file, one should normally also commit the new +resulting generated ``.html`` file(s) into the ``doc/source/sessions-stored/`` +directory at the same time, this ensures that other developers do not need to +regenerate them locally in order to build the docs. + +**Example**: + +.. code:: yaml + + commands: + + # Make it fetch first + - directory: ../examples/foo + command: source fetch hello.bst + + # Capture a build output + - directory: ../examples/foo + output: ../source/sessions/foo-build.html + command: build hello.bst diff --git a/_sources/hacking/writing_plugins.rst.txt b/_sources/hacking/writing_plugins.rst.txt new file mode 100644 index 000000000..7e4ed460f --- /dev/null +++ b/_sources/hacking/writing_plugins.rst.txt @@ -0,0 +1,65 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _writing_plugins: + +Adding core plugins +------------------- +This is a checklist of things which need to be done when adding a new +core plugin to BuildStream proper. + + +Update documentation index +~~~~~~~~~~~~~~~~~~~~~~~~~~ +The documentation generating scripts will automatically pick up your +newly added plugin and generate HTML, but will not add a link to the +documentation of your plugin automatically. + +Whenever adding a new plugin, you must add an entry for it in ``doc/source/core_plugins.rst``. + + +Add tests +~~~~~~~~~ +Needless to say, all new feature additions need to be tested. For ``Element`` +plugins, these usually need to be added to the integration tests. For ``Source`` +plugins, the tests are added in two ways: + +* For most normal ``Source`` plugins, it is important to add a new ``Repo`` + implementation for your plugin in the ``tests/testutils/repo/`` directory + and update ``ALL_REPO_KINDS`` in ``tests/testutils/repo/__init__.py``. This + will include your new ``Source`` implementation in a series of already existing + tests, ensuring it works well under normal operating conditions. + +* For other source plugins, or in order to test edge cases, such as failure modes, + which are not tested under the normal test battery, add new tests in ``tests/sources``. + + +Extend the cachekey test +~~~~~~~~~~~~~~~~~~~~~~~~ +For any newly added plugins, it is important to add some new simple elements +in ``tests/cachekey/project/elements`` or ``tests/cachekey/project/sources``, +and ensure that the newly added elements are depended on by ``tests/cachekey/project/target.bst``. + +One new element should be added to the cache key test for every configuration +value which your plugin understands which can possibly affect the result of +your plugin's ``Plugin.get_unique_key()`` implementation. + +This test ensures that cache keys do not unexpectedly change or become incompatible +due to code changes. As such, the cache key test should have full coverage of every +YAML configuration which can possibly affect cache key outcome at all times. + +See the ``src/buildstream/testing/_update_cachekeys.py`` file for instructions on running the updater, +you need to run the updater to generate the ``.expected`` files and add the new +``.expected`` files in the same commit which extends the cache key test. diff --git a/_sources/handling-files/composition.rst.txt b/_sources/handling-files/composition.rst.txt new file mode 100644 index 000000000..25111924d --- /dev/null +++ b/_sources/handling-files/composition.rst.txt @@ -0,0 +1,207 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _handling_files_composition: + +Composition +=========== +In this chapter we will explore how to create *compositions* of multiple +input filesystem trees, using the :mod:`compose ` element. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/composition + `_ + subdirectory. + + +Overview +-------- +Composing a directory tree based on a set of build dependencies is often +one of the important steps you might perform in order to create a single artifact +which can be checked out and deployed. + +In order to use the :mod:`compose ` element, it is important +to first understand the concept of :ref:`split rules `, which +we will cover in this chapter. + + +Introducing split rules +~~~~~~~~~~~~~~~~~~~~~~~ +The :ref:`split rules ` of an element declaration denote +which sets of files in the given element's resulting artifact belong to which +*domain name*. + +The *domains* can then be used in various ways, using plugins which understand +*split rule domains*. + +BuildStream's :ref:`default project configuration ` +contains a sensible set of default *split rule domains* for the purpose of +artifact splitting, they can be overridden in :ref:`your project.conf `, +and finally on a per element basis in the :ref:`public data ` +of your element declarations. + +.. note:: + + Projects are free to add additional *split rule domains* on top of the + default domains provided by the default project configuration. + + There is nothing wrong with defining split rule domains which *overlap*, + possibly capturing some of the same files also captured by another + *domain*, however you should be aware of this when later using your + split rules with a plugin which processes them, like the + :mod:`compose ` element described in this chapter. + + +Example of split rule declaration +''''''''''''''''''''''''''''''''' +In an element, you might need to define or extend the ``split-rules`` +in order to capture files in custom locations in a logical *domain*. + +Here is an example of how you might use the +:ref:`list append directive ` +to append an additional rule to your ``split-rules`` list in order to +capture additional data files which your application or library might +want to include in the *runtime domain*: + +.. code:: yaml + + # Add our .dat files to the runtime domain + public: + bst: + split-rules: + runtime: + (>): + - | + %{datadir}/foo/*.dat + +Split rules are absolute paths which denote files within an artifact's root +directory. The globbing patterns supported in split rules are defined in the +:func:`reference documentation here `. + +.. important:: + + Note that because of variable expansion, split rules can often be + *resolved differently* for elements which have overridden path + related variables, like ``%{prefix}``. + + This usually means that you do not need to explicitly extend or override + split rules on a specific element unless your element installs files to + special case locations. + + +Project structure +----------------- +In this example we expand on the chapter about +:ref:`integration commands `, so we will +only discuss the files which are added or changed from that example. + + +``elements/base/alpine.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/composition/elements/base/alpine.bst + :language: yaml + +Here we have modified the base runtime, so as to specify that for this +element, we want to also include the runtime linker into the *runtime domain*. + + +``elements/runtime-only.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/composition/elements/runtime-only.bst + :language: yaml + +As we can see, this :mod:`compose ` element has +been configured to only include files from the *runtime domain*. + + +Using the project +----------------- +Now that we've presented how :ref:`split rules ` +work and shown how to use them in the context of this example, lets +use the :mod:`compose ` element we've created and +observe the results. + + +Building the project +~~~~~~~~~~~~~~~~~~~~ + +.. raw:: html + :file: ../sessions/composition-build.html + +As you can see in the output, this composition has only a few hundred +files, but the complete ``alpine.bst`` runtime has several thousand +files. + + +List the content +~~~~~~~~~~~~~~~~ +At the risk of this being a long list, let's :ref:`list the +contents of this artifact ` + +.. raw:: html + :file: ../sessions/composition-list-contents.html + +Some things to observe here: + +* The list does include the ``/usr/bin/hello`` program and + also the ``/usr/lib/libhello.so`` shared library. + + These paths are both captured by the default split rules + for the *runtime domain*. + +* The list does not include the ``/usr/include/libhello.h`` + header file which was used to compile ``/usr/bin/hello``. + + The header file is not captured by the *runtime domain* + by default. It is however captured by the *devel domain*. + +* The runtime linker ``/lib/ld-musl-x86_64.so.1``, as this was + explicitly added to the *runtime domain* for the ``base/alpine.bst`` + element which provides this file. + +.. tip:: + + The reader at this time might want to list the content of + other elements built from this project, such as the + ``hello.bst`` element by itself, or the ``base/alpine.bst`` + element. + + +Run the program +~~~~~~~~~~~~~~~ +Finally, lets just run the program we built. + +.. raw:: html + :file: ../sessions/composition-shell.html + +Here we can see that we at least have the required files to run +our hello world program, however we would not have if we were +missing the runtime linker which we added in ``base/alpine.bst``. + + +Summary +------- +In this chapter we've gotten familiar with :ref:`split rules ` +annotations, and we've learned enough about the :mod:`compose ` +element such that we can start creating our own compositions using +*split domains*. + +We've also used the :ref:`list append directive ` +and we are now observing the contents of artifacts using +:ref:`bst artifact list-contents `. diff --git a/_sources/handling-files/filtering.rst.txt b/_sources/handling-files/filtering.rst.txt new file mode 100644 index 000000000..c030c02d9 --- /dev/null +++ b/_sources/handling-files/filtering.rst.txt @@ -0,0 +1,216 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _handling_files_filtering: + +Filtering +========= +In this chapter we will explore how to *filter* the files in an artifact +using the :mod:`filter ` element, such that an element might +depend on a subset of the files provided by a filtered element. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/filtering + `_ + subdirectory. + + +Overview +-------- +In some cases, it can be useful to depend on a *subset* of the files of an +element, without depending on the entire element. + +One scenario where filtering can be useful, is when you have an element which +will build differently depending on what is present in the system where it is +building. In an edge case where a module fails to offer configure time options to +disable an unwanted feature or behavior in the build, you might use +:mod:`filter ` elements to ensure that special header files or +pkg-config files are *filtered out* from the system at build time, such that +the unwanted behavior cannot be built. + +In many ways, a :mod:`filter ` element is like a +:mod:`compose ` element, except that it operates on a single +:ref:`build dependency `, without compositing the filtered +element with its :ref:`runtime dependencies `. + +.. tip:: + + The :mod:`filter ` element is special in the sense + that it acts as a *window* into it's primary + :ref:`build dependency `. + + As such, :ref:`opening a workspace ` on a + :mod:`filter ` element will result in opening a + workspace on the element which it filters. Any other workspace + commands will also be forwarded directly to the filtered element. + + +Project structure +----------------- +This example again expands on the example presenting in the chapter about +:ref:`integration commands `. In this case +we will modify ``libhello.bst`` such that it produces a new file which, +if present, will affect the behavior of it's reverse dependency ``hello.bst``. + +Let's first take a look at how the sources have changed. + + +``files/hello/Makefile`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/filtering/files/hello/Makefile + :language: Makefile + +Now we have our Makefile discovering the system defined default +person to say hello to. + + +``files/hello/hello.c`` +~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/filtering/files/hello/hello.c + :language: c + +If this program has been given a ``DEFAULT_PERSON``, then it will +say hello to that person in the absence of any argument. + + +``project.conf`` +~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/filtering/project.conf + :language: yaml + +Here, we've added a :ref:`project option ` to decide +whether to use the :mod:`filter ` element or not. + +This is merely for brevity, so that we can demonstrate the behavior +of depending on the filtered element without defining two separate versions +of the ``hello.bst`` element. + + +``elements/libhello.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/filtering/elements/libhello.bst + :language: yaml + +We've added some :ref:`split rules ` here to declare +a new *split domain* named ``defaults``, and we've added the new +``default-person.txt`` file to this *domain*. + + +``elements/libhello-filtered.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/filtering/elements/libhello-filtered.bst + :language: yaml + +And we've added a new :mod:`filter ` element to the project +which uses the ``exclude`` option of the filter configuration. + +This is essentially a statement that any files mentioned in the the +``defaults`` *domain* of the ``libhello.bst`` element should be excluded from +the resulting artifact. + +.. important:: + + Notice that you need to explicitly declare any + :ref:`runtime dependencies ` which are required by the + resulting artifact of a :mod:`filter ` element, as runtime + dependencies of the build dependency are not transient. + + +``elements/hello.bst`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/filtering/elements/hello.bst + :language: yaml + +Here we've merely added a :ref:`conditional statement ` +which allows us to test the ``hello.bst`` element depending on the filtered +version of the library, or the unfiltered version. + + +Using the project +----------------- +Let's just skip over building the ``hello.bst`` element with the +``use_filter`` option both ``True`` and ``False``, these elements +are easily built with :ref:`bst build ` as such: + +.. code:: shell + + bst --option use_filter True build hello.bst + bst --option use_filter False build hello.bst + + +Observing the artifacts +~~~~~~~~~~~~~~~~~~~~~~~ +Let's take a look at the built artifacts. + + +``libhello.bst`` +'''''''''''''''' + +.. raw:: html + :file: ../sessions/filtering-list-contents-libhello.html + +Here we can see the full content of the ``libhello.bst`` artifact. + + +``libhello-filtered.bst`` +''''''''''''''''''''''''' + +.. raw:: html + :file: ../sessions/filtering-list-contents-libhello-filtered.html + +Here we can see that the ``default-person.txt`` file has been filtered +out of the ``libhello.bst`` artifact when creating the ``libhello-filtered.bst`` +artifact. + + +Running hello.bst +~~~~~~~~~~~~~~~~~ +Now if we run the program built by ``hello.bst`` in either build +modes, we can observe the expected behavior. + + +Run ``hello.bst`` built directly against ``libhello.bst`` +''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +.. raw:: html + :file: ../sessions/filtering-shell-without-filter.html + +Here we can see that the hello world program is using the system +configured default person to say hello to. + + +Run ``hello.bst`` built against ``libhello-filtered.bst`` +''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +.. raw:: html + :file: ../sessions/filtering-shell-with-filter.html + +And now we're reverting to the behavior we have when no +system configured default person was installed at build time. + + +Summary +------- +In this chapter, we've introduced the :mod:`filter ` +element which allows one to filter the output of an element and +effectively create a dependency on a subset of an element's files. diff --git a/_sources/handling-files/overlaps.rst.txt b/_sources/handling-files/overlaps.rst.txt new file mode 100644 index 000000000..20ebfca8f --- /dev/null +++ b/_sources/handling-files/overlaps.rst.txt @@ -0,0 +1,158 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _handling_files_overlaps: + +Overlapping files +================= +In this chapter, we will discuss what happens when files from multiple +element artifacts conflict with eachother, and what strategies we can +use to resolve these situations. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/overlaps + `_ + subdirectory. + + +Overview +-------- +This project builds on the previous chapter on :ref:`composition `, +and as such we'll only go over what has changed from there, which is not much. + + +Project structure +----------------- +In this example we've just extended the ``libhello.bst`` and the ``hello.bst`` +elements such that they both install an additional file: ``%{docdir}/hello.txt``. + +We've updated the following Makefiles: + + +files/libhello/Makefile +~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/overlaps/files/libhello/Makefile + :language: Makefile + + +files/hello/Makefile +~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/overlaps/files/hello/Makefile + :language: Makefile + + +As you can see, this now presents a conflict of sorts, where multiple +elements in the pipeline want to install the *same file*. + + +Using the project +----------------- +In this chapter, we're only going to present the warning and then +discuss how to mitigate this situation. + +See what happens when we try to build the ``runtime-only.bst`` +:mod:`compose ` element: + +.. raw:: html + :file: ../sessions/overlaps-build.html + +Notice the warning message about the conflicting file, it is there to +inform the user about which files are *overlapping*, and also which +elements are being staged in which order. + +Note also that BuildStream does not discover the overlap until the +moment that you build a reverse dependency which will require staging +of both artifacts. + +.. tip:: + + The ``overlaps`` warning discussed here can be configured to be + a :ref:`fatal warning `. This is useful + in the case that you want to be strict about avoiding overlapping + files in your project. + + +Mitigating overlapping files +---------------------------- +Since we recently discussed :ref:`filtering of artifacts `, +we should note that it is of course possible to handle this case by +having ``hello.bst`` depend on a *filtered* version of ``libhello.bst`` with the +offending file excluded. + +However, working with :mod:`filter ` elements just for the +sake of handling a conflicting artifact would be quite inconvenient, so we +have other means. + + +Whitelisting overlapping files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +BuildStream allows explicitly ignoring such errors by adding the files +to an :ref:`overlap whitelist `, you could achieve +this in the given example by adding the following to the ``hello.bst`` element: + +.. code:: yaml + + public: + bst: + overlap-whitelist: + - | + %{docdir}/hello.txt + +.. note:: + + Note that :func:`glob patterns ` are also + supported in the whitelist. + + +Artifact munging +~~~~~~~~~~~~~~~~ +Another way around this situation is the *munge* the artifacts at install +time such that there is no conflict. + +This is the easiest approach in the case that you might want to keep the +underlying ``%{docdir}/hello.txt`` from ``libhello.bst`` and discard the +same file from ``hello.bst``. + +In this case, we might modify the ``hello.bst`` file so that it's install +command contain an ``rm`` statement, as such: + +.. code:: yaml + + install-commands: + - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install + + - | + # Rid ourselves of the unwanted file at install time + rm -f %{install-root}%{docdir}/hello.txt + +This would cause later builds of ``runtime-only.bst`` to no longer +conflict on the given file. + + +Summary +------- +In this chapter we've presented a situation where an artifact +can *conflict* with another artifact by way of providing the +same files. + +We've presented the :ref:`overlap whitelist ` +public data which is the typical solution for silencing the +error when the outcome is desired, and also presented a strategy +to deal with cases where you want to keep files from the +*overlapped* artifact instead. diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt new file mode 100644 index 000000000..e0be32484 --- /dev/null +++ b/_sources/index.rst.txt @@ -0,0 +1,37 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +BuildStream documentation +========================= +These docs cover everything you need to build and integrate software stacks +using BuildStream. + +They begin with a basic introduction to BuildStream, background +information on basic concepts, and a guide to the BuildStream command line interface. +Later sections provide detailed information on BuildStream internals. + +.. toctree:: + :maxdepth: 1 + + main_about + main_install + main_using + main_core + main_porting.rst + CONTRIBUTING + main_architecture + main_glossary + +For any other information, including `how to install BuildStream `_, +refer to `the BuildStream website `_. diff --git a/_sources/junctions/junction-elements.rst.txt b/_sources/junctions/junction-elements.rst.txt new file mode 100644 index 000000000..f0effec88 --- /dev/null +++ b/_sources/junctions/junction-elements.rst.txt @@ -0,0 +1,105 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _junction_elements: + +Junction elements +================= +BuildStream's :mod:`junction ` elements are the mechanism which +allow projects to interact and depend on eachother. + +Junction elements represent the BuildStream project you are depending, and behave +much like other elements in the sense that they can be :ref:`fetched ` +and :ref:`tracked ` like other elements, except that regular +elements cannot *depend* on junctions directly, nor can junctions be :ref:`built `. +Instead, junctions act like a window into another project you depend on, and allow +elements of your project to depend on elements exposed by the project referenced by +the junction. + +Projects which are junctioned by your project are referred to as *subprojects*. + + +A simple example +---------------- + +.. note:: + + This example is distributed with BuildStream in the + `doc/examples/junctions `_ + subdirectory. + +Below is a simple example of bst file for the junction element, which +we have called ``hello-junction.bst`` in this project: + +.. literalinclude:: ../../examples/junctions/elements/hello-junction.bst + :language: yaml + +This element imports the autotools example subproject distributed with BuildStream +in the `doc/examples/junctions/autotools `_ +subdirectory. + +.. note:: + + For the sake of this example we are using a local source in a subdirectory + of the example project. + + Since junctions allow interoperability of projects, it would be more common + to use a junction to a remote project under separate revision control, possibly + using a ``kind: git`` source. + +The below bst file describes the element ``callHello.bst``, which depends on the +``hello.bst`` element from the autotools example: + +.. literalinclude:: ../../examples/junctions/elements/callHello.bst + :language: yaml + +Note how this element refers to the previously declared ``hello-junction.bst`` +junction in its :ref:`dependency dictionary `. This dependency +expresses that we are depending on the ``hello.bst`` element in the project +which ``hello-junction.bst`` refers to. + +The ``callHello.bst`` element simply imports a ``callHello.sh`` shell script which +calls the hello command provided by ``hello.bst``: + +.. literalinclude:: ../../examples/junctions/files/callHello.sh + :language: shell + + +Building and running +-------------------- +Building the ``callHello.bst`` element which requires an external project +is just a matter of invoking :ref:`bst build ` in the +regular way: + +.. raw:: html + :file: ../sessions/junctions-build.html + +You can see that the hello.bst element and its dependencies from the autotools +project have been built as a part of the pipeline for callHello.bst. + +We can now invoke :ref:`bst shell ` and run our ``callHello.sh`` +script, which in turn also calls the ``hello`` program installed by the +subproject's ``hello.bst`` element. + +.. raw:: html + :file: ../sessions/junctions-shell.html + + +Further reading +--------------- +For an example of junction elements being used in a real project, take a look +at the `freedesktop-sdk junction +`_ +in the `gnome-build-meta `_ project. diff --git a/_sources/junctions/junction-includes.rst.txt b/_sources/junctions/junction-includes.rst.txt new file mode 100644 index 000000000..0f588059b --- /dev/null +++ b/_sources/junctions/junction-includes.rst.txt @@ -0,0 +1,264 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _advanced_junction_includes: + +Subproject includes +=================== +We've already discussed how we can add optionality to projects and +explored how we can perform conditional statements and include fragments +of BuildStream YAML in the earlier :ref:`chapter about optionality and +directives `. + +In this chapter we're going to explore how we can use :ref:`include directives +` to include YAML fragments from a subproject +referred to by a :mod:`junction ` element, and how +:ref:`project options ` can be specified in the configuration +of your :mod:`junction `. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/junction-includes + `_ + subdirectory. + + +Overview +-------- +It is a goal of BuildStream to provide developers and integrators with the tools +they need to maintain software stacks which depend on eachother with least friction +as possible, such that one can integrate upgrades of projects one depends on +via :mod:`junction ` elements regularly and with the least +hassle as possible. + +:ref:`Project options ` and :ref:`include directives +` combined form the basis on which projects can +maximize on code sharing effectively, and the basis on which BuildStream +projects can form reliable APIs. + + +Project options +~~~~~~~~~~~~~~~ +The :ref:`options ` which a project exposes is a fairly +limited API surface, it allows one to configure a limited set of options +advertized by the project maintainers, and the options will affect what +kind of artifacts will be produced by the project. + +This kind of optionality however does not allow consumers to entirely +redefine how artifacts are produced and how elements are configured. + +On the one hand, this limitation can be frustrating, as one constantly +finds themselves requiring a feature that their subproject does not +support *right now*. On the other hand, the limitation of features which +a given project chooses to support is what guards downstream project +consumers against consuming artifacts which are not supported by the upstream. + +Project options are designed to enforce a *separation of concerns*, +where we expect that downstreams will either fork a project in order +to support a new feature, or convince the upstream to start supporting +a new feature. Furthermore, limited API surfaces for interdependent +projects offers a possibility of API stability of projects, such +that you can upgrade your dependencies with limited friction. + + +Includes +~~~~~~~~ +The :ref:`includes ` which a project might advertize +as *"public"*, form the output of the API exchange between a project and +its subproject(s). + +Cross-project include files allow a project to *inherit configuration* from +a subproject. Include files can be used to define anything from the +:ref:`variables ` one needs to have in context in order to +build into or link into alternative system prefixes, what special compiler flags +to use when building for a specific machine architecture, to customized +:ref:`shell configurations ` to use when testing out applications +in :ref:`bst shell `. + +This chapter will provide an example of the *mechanics* of cross project +includes when combined with project optionality. + + +Project structure +----------------- + + +Project options +~~~~~~~~~~~~~~~ +This example is comprised of two separate projects, both of which offer +some project options. This is intended to emphasize how your toplevel project +options can be used to select and configure options to use in the subprojects +you depend on. + +For convenience, the subproject is stored in the subdirectory of +the toplevel project, while in the real world the subproject is probably +hosted elsewhere. + +First let's take a look at the options declared in the respective +``project.conf`` files. + + +Toplevel ``project.conf`` +''''''''''''''''''''''''' + +.. literalinclude:: ../../examples/junction-includes/project.conf + :language: yaml + + + +Subproject ``project.conf`` +''''''''''''''''''''''''''' + +.. literalinclude:: ../../examples/junction-includes/subproject/project.conf + :language: yaml + + +As we can see, these two projects both offer some arbitrarily named options. + + +Conditional configuration of subproject +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The toplevel project here does some conditional configuration of the +subproject. + + +Toplevel ``elements/subproject-junction.bst`` +''''''''''''''''''''''''''''''''''''''''''''' + +.. literalinclude:: ../../examples/junction-includes/elements/subproject-junction.bst + :language: yaml + +Here we can see that projects can use +:ref:`conditional statements ` to make +decisions about subproject configuration based on their own configuration. + +In this example, if the toplevel project is ``funky``, then it will +configure its subproject with ``color`` set to ``blue``, otherwise it +will use the ``red`` variant of the subproject ``color``. + + +Including configuration from a subproject +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Here there are a couple of aspects to observe, namely how the +toplevel project includes files across a junction boundary, +and how that include file might be implemented. + + +Toplevel ``elements/hello.bst`` +''''''''''''''''''''''''''''''' + +.. literalinclude:: ../../examples/junction-includes/elements/hello.bst + :language: yaml + +Here we can see the same element which we discussed in the +:ref:`autotools example `, except that we're including +a file from the subproject. As explained in the :ref:`reference manual `, +this is done by prefixing the include path with the local :mod:`junction ` +element name and then a colon. + +Note that in this case, the API contract is simply that ``hello.bst`` is +including ``paths.bst``, and has the expectation that ``paths.bst`` will +in some way influence the ``variables``, nothing more. + +It can be that an include file is expected to create new variables, and +it can be that the subproject might declare things differently depending +on the subproject's own configuration, as we will observe next. + + +Subproject ``include/paths.bst`` +'''''''''''''''''''''''''''''''' + +.. literalinclude:: ../../examples/junction-includes/subproject/include/paths.bst + :language: yaml + +Here, we can see the include file *itself* is making a +:ref:`conditional statement `, in turn +deciding what values to use depending on how the project was configured. + +This decision will provide valuable context for any file including ``paths.bst``, +whether it be an element, a ``project.conf`` which applies the variable as +a default for the entire project, whether it is being included by files +in the local project, or whether it is being included by a downstream +project which junctions this project, as is the case in this example. + + +Using the project +----------------- +At this stage, you have probably already reasoned out what would happen +if we tried to build and run the project. + +Nevertheless, we will still present the outputs here for observation. + + +Building the project normally +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Here we build the project without any special arguments. + +.. raw:: html + :file: ../sessions/junction-includes-build-normal.html + + +Building the project in funky mode +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Now let's see what happens when we build the project in funky mode + +.. raw:: html + :file: ../sessions/junction-includes-build-funky.html + +As we can see, this time we've built the project into the ``/opt`` +system prefix instead of the standard ``/usr`` prefix. + +Let's just take a step back now and summarize the process which +went into this decision: + +* The toplevel ``project.conf`` exposes the boolean ``funky`` option + +* The toplevel junction ``subproject-junction.bst`` chooses to set the + subproject ``color`` to ``blue`` when the toplevel project is ``funky`` + +* The subproject ``include/paths.bst`` include file decides to set the + ``prefix`` to ``/opt`` in the case that the subproject is ``blue`` + +* The ``hello.bst`` includes the ``include/paths.bst`` file, in order + to inherit its path configuration from the subproject + + +Running the project in both modes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. raw:: html + :file: ../sessions/junction-includes-shell-normal.html + +.. raw:: html + :file: ../sessions/junction-includes-shell-funky.html + +As expected, the ``funky`` variant of the toplevel project installs +the hello world program in the ``/opt`` prefix, and as such we +need to call it from there. + + +Summary +------- +In this chapter we've discussed how :ref:`conditional statements ` +and :ref:`include files ` play an essential role +in the API surface of a project, and help to provide some configurability +while preserving encapsulation of the API which a project exposes. + +We've also gone over the mechanics of how these concepts interact and +presented an example which shows how project options can be used in +a recursive context, and how includes can help not only to share code, +but to provide context to dependent projects about how their subprojects +are configured. diff --git a/_sources/junctions/junction-workspaces.rst.txt b/_sources/junctions/junction-workspaces.rst.txt new file mode 100644 index 000000000..23ebe3e50 --- /dev/null +++ b/_sources/junctions/junction-workspaces.rst.txt @@ -0,0 +1,64 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _junction_workspaces: + +Workspaces and subprojects +========================== +When developping a project with :mod:`junctions ` and +subprojects, you will probably end up needing to work on the subprojects +as well. + +Opening :ref:`workspaces ` works mostly in the +same way as it does with subprojects as it does for elements directly +in your own project. + +.. note:: + + This section runs commands on the same example project presented in the + :ref:`previous section `, which is distributed with BuildStream in the + `doc/examples/junctions `_ + subdirectory. + + +Workspacing a junction +---------------------- +Sometimes you need to work on the elements declared in a subproject +directly. As the downstream consumer of a junctioned project, it makes +sense that you might need to work on that project as well in order +to satisfy the needs of your downstream project. + +You can easily work on your subproject by :ref:`opening a workspace ` +on the junction element directly. + +.. raw:: html + :file: ../sessions/junctions-workspace-open-subproject.html + +After opening a workspace on the junction element, the open workspace +is used to define the subproject, allowing you to make changes to +how the subproject is built, add new dependencies and configure the +subproject in any way. + + +Cross-junction workspaces +------------------------- +You can open workspaces for elements in the project refered to by the junction +using the syntax ``bst open ${junction-name}:{element-name}``. In this example, + +.. raw:: html + :file: ../sessions/junctions-workspace-open.html + +This has opened a workspace for the hello.bst element from the autotools project. +This workspace can now be used as normal. diff --git a/_sources/main_about.rst.txt b/_sources/main_about.rst.txt new file mode 100644 index 000000000..eb8c037ef --- /dev/null +++ b/_sources/main_about.rst.txt @@ -0,0 +1,14 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. include:: ../../README.rst diff --git a/_sources/main_architecture.rst.txt b/_sources/main_architecture.rst.txt new file mode 100644 index 000000000..cff4d7428 --- /dev/null +++ b/_sources/main_architecture.rst.txt @@ -0,0 +1,33 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Architecture +============ +This section provides details on the overall BuildStream architecture. + + +.. toctree:: + :maxdepth: 2 + + arch_overview + arch_program_flow + arch_data_model + arch_dependency_model + arch_scheduler + arch_cachekeys + arch_caches + arch_sandboxing + arch_remote_execution + diff --git a/_sources/main_core.rst.txt b/_sources/main_core.rst.txt new file mode 100644 index 000000000..0049de0a1 --- /dev/null +++ b/_sources/main_core.rst.txt @@ -0,0 +1,39 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Reference +========= +This section details the core API reference along with +other more elaborate details about BuildStream internals. + + +.. toctree:: + :maxdepth: 2 + + core_format + core_plugins + core_framework + core_additional + + +.. + This is a hidden toctree so that the autogenerated + modules index is not orphaned, sort of a cheat because + we would rather present a manual toctree for this part. + +.. toctree:: + :hidden: + + buildstream diff --git a/_sources/main_glossary.rst.txt b/_sources/main_glossary.rst.txt new file mode 100644 index 000000000..f16fac186 --- /dev/null +++ b/_sources/main_glossary.rst.txt @@ -0,0 +1,165 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Glossary +======== + +.. glossary:: + :sorted: + + + ``.bst`` file + The configuration for an :term:`Element `, represented + in YAML format. + + + Artifact + The output collected after building an :term:`Element`. + + Artifacts can be built from :term:`Sources `, or pulled from a + :term:`Remote Cache `, if available. + + Artifact name + The :ref:`name of an artifact `, which can be used + in various commands to operate directly on artifacts, without requiring + the use of a :term:`Project`. + + Cache + BuildStream leverages various caching techniques in order to avoid + duplicating work. + + Depending on context, "Cache" might refer to BuildStream's :term:`local + cache ` or a :term:`Remote Cache `. + + + Core plugin + A :term:`Plugin ` that is contained in the BuildStream + package. These are built-in and don't need to be defined in the + project configuration. + + See :ref:`plugin documentation ` for more details on core + plugins. + + + Dependency + :term:`Elements ` in a BuildStream project can depend + on other elements from the same project. The element dependent upon is + called a dependency. + + See :ref:`Dependencies document ` for more + details. + + Dependency configuration + Additional custom YAML configuration which is used to define + an :term:`Element's ` relationship with it's :term:`Dependency `. + + This is supported on limited :term:`Element ` implementations, and + each :term:`Element ` defines what configuration it supports. + + See the :ref:`dependency documentation ` for details + on dependency configuration. + + Element + An atom of a :term:`BuildStream project `. Projects consist of + zero or more elements. + + During the build process, BuildStream transforms :term:`Sources + ` and :term:`Dependencies ` of an + element into its output. The output is called an + :term:`Artifact `. + + Configuration for elements is stored in form of :term:`.bst files + <.bst file>`. See :ref:`Declaring Elements document ` + for more details on element configurtion. + + + External Plugin + A :term:`Plugin ` that is defined in some package other + than BuildStream. + + External plugins must be declared in :ref:`the project configuration + `. + + For a list of known external plugin repositories, see + :ref:`plugins_external`. + + + Junction + A special kind of :term:`Element `, that allows you to + depend on elements from another project. + + See :mod:`Junction reference ` for details on how to + configure junction elements. + + See :ref:`Junction guide ` for details on how to use + junction elements. + + + Local Cache + To avoid duplicating work, BuildStream will cache sources, artifacts, + logs, buildtrees etc. in a local cache directory. If these sources or + artifacts are needed another time, BuildStream will use them from the + cache. + + See :ref:`Local cache expiry ` section of the user + guide for details on how to configure the local cache. + + + Plugin + BuildStream Plugins define types of :term:`Elements ` + and :term:`Sources `. Hence, they come in two distinct + varities - Element Plugins and Source Plugins. + + BuildStream supports some plugins :term:`out of the box + `. It also has support for :term:`third party + plugins `. + + + Project + A collection of :term:`Elements `. + + Elements in a project share some central configuration. See + :ref:`projectconf` to learn how to configure BuildStream projects. + + + Remote Cache + A server setup for sharing BuildStream :term:`Sources ` + and/or :term:`Artifacts `. + + See :ref:`cache server documentation ` for details on + artifact caches. + + + Source + Sources describe the input to the build of an :term:`Element`. + + In general, an element can have zero or more sources. But, certain + element plugins may restrict the number of allowed sources. + + Sources are defined in the :ref:`Sources ` section of + :term:`Element ` configuration. + + + Subproject + Subprojects are :term:`projects ` which are referred + to by a :term:`Junction`. + + + Workspace + Workspaces allow building one or more elements using a local, and + potentially modified, copy of their sources. + + See :ref:`Workspaces guide ` for more details on + how to use workspaces. diff --git a/_sources/main_install.rst.txt b/_sources/main_install.rst.txt new file mode 100644 index 000000000..edc0abb0d --- /dev/null +++ b/_sources/main_install.rst.txt @@ -0,0 +1,286 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +Installing from Source +====================== + +This page explains how to build and install this version of BuildStream from +source. For general purpose installation instructions consult the +`website `_. + +For full install instructions, read on: + +* :ref:`Install dependencies` +* :ref:`Install BuildBox` +* :ref:`Install BuildStream` +* :ref:`Install completions` + +.. _install-dependencies: + +Installing Dependencies +----------------------- + +Runtime requirements +~~~~~~~~~~~~~~~~~~~~ + +BuildStream requires the following Python environment to run: + +- python3 >= 3.8 +- PyPI packages as specified in + `requirements.in `_. + +Some :mod:`Source ` plugins require specific tools installed +on the host. Here is a commonly used subset based on the +:ref:`core source plugins ` +and `buildstream-plugins `_. + +- git (for ``git`` sources) +- lzip (for ``.tar.lz`` support in ``tar`` sources) +- patch (for ``patch`` sources) + +Some BuildBox tools used by BuildStream require additional host tools: + +- bubblewrap (for ``buildbox-run-bubblewrap``) +- fusermount3 (for ``buildbox-fuse``) + +Install-time requirements +~~~~~~~~~~~~~~~~~~~~~~~~~ + +BuildStream contains Cython code which implies the following extra +dependencies at install-time only: + +- C and C++ toolchain +- Python development headers + +These instructions use ``pip3`` to install necessary PyPI packages. +Packagers and integrators may use a different tool and can ignore +the `pip` dependency below. + +Distribution-specific guides +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This table gives you a list of packages for specific distros: + +.. list-table:: + + * - **Distribution** + - **Runtime requires** + - **Install requires** + * - Arch Linux + - bubblewrap fuse3 git lzip patch python + - gcc python-pip + * - Debian + - bubblewrap fuse3 git lzip patch python3 + - g++ python3-dev python3-pip + * - Fedora + - bubblewrap fuse3 git lzip patch python3 + - gcc-c++ python3-devel python3-pip + * - Ubuntu + - bubblewrap fuse3 git lzip patch python3 + - g++ python3-dev python3-pip + +.. _install-buildbox: + +Installing BuildBox +------------------- + +BuildStream depends on the following tools from +`BuildBox `_: + + * ``buildbox-casd`` (to manage local and remote content-addressed storage) + * ``buildbox-fuse`` (to check out content from the local CAS) + * ``buildbox-run-bubblewrap`` (to run element commands in a controlled sandbox) + +These components can be installed from binaries, or built from source. + +Install binaries +~~~~~~~~~~~~~~~~ +Browse the `release history of static binaries here +`_. + +Linux x86-64 users can download the `latest statically linked binaries here +`_, +The contents of the tarball should be extracted into a directory in ``PATH``, +e.g., ``~/.local/bin``. + + +Build from source +~~~~~~~~~~~~~~~~~ + +Buildbox components can be installed from their git repository. +We recommend installing from the latest release tag. + +See the <"Installation" section here `_. + +Buildbox contains many components, BuildStream needs only ``buildbox-casd``, +``buildbox-fuse`` and ``buildbox-run-bubblewrap``, which can be selected by +passing options ``-DTOOLS=OFF -DCASD=ON -DFUSE=ON -DRUN_BUBBLEWRAP=ON`` to CMake. + +Finally, configure buildbox-run-bubblewrap as the default buildbox-run +implementation:: + + ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run + + + +.. _install-buildstream: + +Installing BuildStream from a git checkout +------------------------------------------ + +First, clone the repository. Please check the existing tags in the git +repository and determine which version you want to install:: + + + git clone https://github.com/apache/buildstream.git + cd buildstream + git checkout + +We recommend ``pip`` as a frontend to the underlying ``setuptools`` build +system. The following command will build and install BuildStream into your +user's homedir in ``~/.local``, and will attempt to fetch and install any +required PyPI dependencies from the internet at the same time:: + + + pip3 install --user . + +We do not recommend using Pip's `editable mode `_ +(the ``-e`` flag). See `this issue `_ for discussion. + +If you want to stop Pip from fetching missing dependencies, use the +``--no-index`` and ``--no-deps`` options. + +Finally, check that the ``PATH`` variable contains the ``~/.local/bin`` directory. +If it doesn't, you could add this to the end of your Bash configuration ``~/.bashrc`` +and restart Bash:: + + export PATH="${PATH}:${HOME}/.local/bin" + +Note for packagers +~~~~~~~~~~~~~~~~~~ + +Distro packaging standards may recommend a specific installation method +for Python packages. BuildStream can be installed with any build frontend that +supports the `PEP517 standard `_. You are +also welcome to use the underlying +`setuptools `_ build backend directly. + + +.. _install-virtual-environment: + +Installing in virtual environments +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +You can consider installing BuildStream in a +`Virtual Environment `_ if you want +to install multiple versions of BuildStream, or to isolate BuildStream and its +dependencies from other Python packages. + +Here is how to install BuildStream stable and development snapshot releases in +virtual environments of their own:: + + + # Install BuildStream stable in an environment called "venv-bst-stable" + # (At time of writing, this will be BuildStream 1) + python3 -m venv venv-bst-stable + venv-bst-stable/bin/pip install BuildStream + + # Install BuildStream latest development snapshot in an environment + # called "venv-bst-latest" + # (At time of writing, this will be Buildstream 2) + python3 -m venv venv-bst-latest + venv-bst-latest/bin/pip install --pre BuildStream + +To start using BuildStream from the desired environment, you will need to +activate it first. Activating it will automatically add ``bst`` to your ``PATH`` +and set up other necessary environment variables:: + + + # Use BuildStream stable from venv-bst-stable + source venv-bst-stable/bin/activate + bst --version + + # Use BuildStream latest from venv-bst-latest + source venv-bst-latest/bin/activate + bst --version + + # Once you are done, remember to deactivate the virtual environment + deactivate + +If you do not want to manage your virtual environments manually, you can +consider using `pipx `_. + +.. _install-completions: + +Installing completions +---------------------- + +BuildStream integrates with Bash and Zsh to provide helpful tab-completion. +These completion scripts require manual installation. + +Bash completions +~~~~~~~~~~~~~~~~ + +Bash completions are provided by the ``bst`` completion script, available online +(`src/buildstream/data/bst `_) +and in your local Git clone at ``src/buildstream/data/bst``. + +To install for the current user, paste the contents of the completion script +into the file ``~/.bash_completion``. + +To install completions system-wide, copy the completion script to the system-wide +bash-completion installation path, which you can discover as follows:: + + pkg-config --variable=completionsdir bash-completion + +See the `bash-completion FAQ `_ +for more information. + +Zsh completions +~~~~~~~~~~~~~~~~ + +Zsh completions are provided by the ``_bst`` completion script, available online +(`src/buildstream/data/zsh/_bst `_) +and in your local Git clone at ``src/buildstream/data/zsh/_bst``. + +Copy the above file to your Zsh completions location. Here are some instructions +for vanilla Zsh, as well as the *Prezto* and *Oh My Zsh* frameworks: + +**Zsh**:: + + cp src/buildstream/data/zsh/_bst ~/.zfunc/_bst + +You must then add the following lines in your ``~/.zshrc``, if they do not already exist:: + + fpath+=~/.zfunc + autoload -Uz compinit && compinit + + +**Prezto**:: + + cp src/buildstream/data/zsh/_bst ~/.zprezto/modules/completion/external/src/_bst + +You may have to reset your zcompdump cache, if you have one, and then restart your shell:: + + rm ~/.zcompdump ${XDG_CACHE_HOME:-$HOME/.cache}/prezto/zcompdump + +**Oh My Zsh**:: + + mkdir $ZSH_CUSTOM/plugins/bst + cp src/buildstream/data/zsh/_bst $ZSH_CUSTOM/plugins/bst/_bst + +You must then add ``bst`` to your plugins array in ``~/.zshrc``:: + + plugins( + bst + ... + ) diff --git a/_sources/main_porting.rst.txt b/_sources/main_porting.rst.txt new file mode 100644 index 000000000..62f534d56 --- /dev/null +++ b/_sources/main_porting.rst.txt @@ -0,0 +1,29 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _main_porting: + +Porting guide +============= +This document focuses on porting your existing BuildStream 1 projects to +using BuildStream 2. + + +.. toctree:: + :maxdepth: 1 + + porting_user_configuration + porting_command_line + porting_project diff --git a/_sources/main_using.rst.txt b/_sources/main_using.rst.txt new file mode 100644 index 000000000..e5dfee408 --- /dev/null +++ b/_sources/main_using.rst.txt @@ -0,0 +1,35 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _using: + +Using +===== +This section includes user facing documentation including tutorials, +guides and information on user preferences and configuration. + + +.. toctree:: + :maxdepth: 2 + + using_tutorial + using_developing + using_handling_files + using_junctions + using_config + using_commands + using_configuring_cache_server + using_configuring_remote_execution + using_examples diff --git a/_sources/porting_command_line.rst.txt b/_sources/porting_command_line.rst.txt new file mode 100644 index 000000000..5b24697aa --- /dev/null +++ b/_sources/porting_command_line.rst.txt @@ -0,0 +1,186 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _porting_command_line: + +Porting command line usage +========================== +This document outlines breaking changes made to the :ref:`command line interface ` +in BuildStream 2. + + +:ref:`bst init ` +------------------------------- + +* The global ``--directory`` option is no longer observed by ``bst init``, instead + the command accepts an optional target directory argument. + +* The ``--format-version`` option has been removed in favor of the new ``--min-version`` option. + + +:ref:`bst build ` +--------------------------------- + +* Tracking is no longer supported at build time and must be performed separately, this + removes the ``--track``, ``--track-all``, ``--track-except``, ``--track-cross-junctions`` + and ``--track-save`` options from the command. + + To track your elements in BuildStream 2, use the :ref:`bst source track ` + command instead. + +* The ``--all`` option which was used to indicate that all dependencies should be built + regardless of whether they are needed for producing the target elements has been removed + in favor of adding the ``--deps`` option. + + To acheive the same functionality, use ``bst build --deps all ...``. + + +:ref:`bst show ` +------------------------------- + +* The ``plan`` value is no longer supported as a value for the ``--deps`` option. + +* Values for the ``%{state}`` format have changed + + * :mod:`junction ` elements will display ``junction``, as these cannot be built + * In the case a cached failed build artifact is found, then ``failed`` will be displayed + * Due to changes in the scheduler, we may observe changes as to when ``waiting``, ``buildable``, ``fetch needed`` + are displayed for a given element. + + +:ref:`bst fetch ` +---------------------------------------- + +* This command has been removed as a top-level command and now exists as :ref:`bst source fetch ` + +* Tracking is no longer supported at fetch time and must be performed separately, this + removes the ``--track`` and ``--track-cross-junctions`` options from the command. + + To track your elements in BuildStream 2, use the :ref:`bst source track ` + command instead. + +* The ``plan`` value is no longer supported as a value for the ``--deps`` option. The default + value for the ``--deps`` option is now ``none``. + + +:ref:`bst track ` +---------------------------------------- + +* This command has been removed as a top-level command and now exists as :ref:`bst source track ` + + +:ref:`bst pull ` +---------------------------------------- + +* This command has been removed as a top-level command and now exists as :ref:`bst artifact pull ` + +* The ``--remote`` option has been removed in favor the ``--artifact-remote`` option, which can be + specified multiple times. + +* The values which can be specified by ``--artifact-remote`` options have a new format which + is :ref:`documented here `. + + +:ref:`bst push ` +---------------------------------------- + +* This command has been removed as a top-level command and now exists as :ref:`bst artifact push ` + +* The ``--remote`` option has been removed in favor the ``--artifact-remote`` option, which can be + specified multiple times. + +* The values which can be specified by ``--artifact-remote`` options have a new format which + is :ref:`documented here `. + + + +:ref:`bst checkout ` +------------------------------------------------ + +* This command has been removed as a top-level command and now exists as :ref:`bst artifact checkout ` + +* The trailing ``LOCATION`` argument has been removed in favor of a ``--directory`` option. + + **BuildStream 1:** + + .. code:: shell + + bst checkout element.bst ~/checkout + + **BuildStream 2:** + + .. code:: shell + + bst artifact checkout --directory ~/checkout element.bst + + +:ref:`bst shell ` +--------------------------------- + +* The ``--sysroot`` option has been completely removed. + + This is no longer needed for failed builds as the build tree will be cached in a failed build artifact. + +* Sources and artifacts required to produce the shell environment will now be downloaded + automatically by default. + + +:ref:`bst workspace open ` +--------------------------------------------------- + +* The ``--track`` option is now removed. + +* The trailing ``LOCATION`` argument has been removed in favor of a ``--directory`` option. + + **BuildStream 1:** + + .. code:: shell + + bst workspace open element.bst ~/workspace + + **BuildStream 2:** + + .. code:: shell + + bst workspace open --directory ~/workspace element.bst + + +:ref:`bst workspace reset ` +----------------------------------------------------- + +* The ``--track`` option is now removed. + + +:ref:`bst source-bundle ` +--------------------------------------------------- +This command has been completely removed, but similar behavior can be achieved +using the :ref:`bst source checkout ` command. + + +**BuildStream 1:** + +.. code:: shell + + bst source-bundle --directory ~/bundle element.bst + +**BuildStream 2:** + +.. code:: shell + + bst source checkout \ + --tar ~/sources.tgz \ + --compression gz \ + --include-build-scripts \ + element.bst diff --git a/_sources/porting_project.rst.txt b/_sources/porting_project.rst.txt new file mode 100644 index 000000000..9a981ca3c --- /dev/null +++ b/_sources/porting_project.rst.txt @@ -0,0 +1,385 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _porting_project: + +Porting the project format +========================== +This document outlines breaking changes made to the project format in BuildStream 2. + + +The project.conf +---------------- +This section outlines breaking changes made to the :ref:`project.conf format `. + + +Project name +~~~~~~~~~~~~ +Various features related to :mod:`junction ` elements have been added +which allow addressing projects by their :ref:`project names `. For this +reason, it is important to ensure that your project names are appropriately unique. + + +Project versioning +~~~~~~~~~~~~~~~~~~ +Instead of maintaining a separate version number for the format and for BuildStream releases, +projects now declare the minimum version of BuildStream 2 they depend on. + +The ``format-version`` attribute should be removed from your project.conf (if present) and +the :ref:`min-version ` attribute must be added. + + +Some attributes can only be specified in project.conf +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Whenever specifying any of the following toplevel project attributes, they must be +specified inside the project.conf itself and cannot be :ref:`included ` +from a separate file: + +* :ref:`name ` +* :ref:`element-path ` +* :ref:`min-version ` +* :ref:`plugins ` + + +Artifact cache configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The format for declaring :ref:`artifact caches ` which are associated to +your project have been completely redesigned. + +**BuildStream 1:** + +.. code:: yaml + + # + # We used to specify a single URL + # + artifacts: + url: https://foo.com/artifacts + +**BuildStream 2:** + +.. code:: yaml + + # + # Now we declare a list, and credentials have been split + # out into a separate "auth" dictionary + # + artifacts: + - url: https://foo.com:11001 + auth: + server-cert: server.crt + + +Loading plugins +~~~~~~~~~~~~~~~ +The format for :ref:`loading plugins ` has been completely redesigned. + +.. tip:: + + A new method for :ref:`loading plugins through junctions ` + has been added. In the interest of ensuring strong determinism and reliability it is + strongly recommended to use this new method. + + +Local plugins +''''''''''''' +Here is an example of how loading :ref:`local plugins ` has changed. + +**BuildStream 1:** + +.. code:: yaml + + plugins: + + - origin: local + path: plugins/sources + + # + # We used to specify version numbers, these no longer exist. + # + sources: + mysource: 0 + +**BuildStream 2:** + +.. code:: yaml + + plugins: + + - origin: local + path: plugins/sources + + # + # Now we merely specify a list of plugins to load from + # a given project local directory + # + sources: + - mysource + + +Pip plugins +''''''''''' +Here is an example of how loading :ref:`pip plugins ` has changed. + +**BuildStream 1:** + +.. code:: yaml + + plugins: + + - origin: pip + + package-name: vegetables + + # + # We used to specify version numbers, these no longer exist. + # + elements: + potato: 0 + +**BuildStream 2:** + +.. code:: yaml + + plugins: + + - origin: pip + + # + # We can now specify version constraints + # + package-name: vegetables>=1.2 + + # + # Now we merely specify a list of plugins to load from + # a given pip package that is expected to be installed + # + elements: + - potato + + +Core elements +------------- +This section outlines breaking changes made to :ref:`core element plugins ` which +may require you to make changes to your project. + + +The :mod:`stack ` element +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Stack elements dependencies are now hard required to be both build and runtime dependencies. + + +The :mod:`script ` element +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The ``layout`` attribute has now been removed in favor of dependency level configuration. + +Here is an example script which collects a manifest of all files in the hypothetical +``system.bst`` element, using a hypothetical base runtime element ``base-utilities.bst``. + +**BuildStream 1:** + +.. code:: yaml + + kind: script + + build-depends: + - base-utilities.bst + - system.bst + + config: + # + # The old format was redundant and required explicit layout + # of the dependencies already declared above. + # + layout: + - element: base-utilities.bst + destination: / + - element: system.bst + destination: "%{build-root}" + + commands: + - find %{build-root} > %{install-root}/manifest.log + +**BuildStream 2:** + +.. code:: yaml + + kind: script + + # + # The default location is "/" so there is no need to configure + # the "base-utilities.bst" dependency + # + build-depends: + - base-utilities.bst + - system.bst + config: + location: "%{build-root}" + + config: + commands: + - find %{build-root} > %{install-root}/manifest.log + +.. tip:: + + The ``location`` dependency level configuration is also supported by all + :mod:`BuildElement ` plugins. + + +The :mod:`junction ` element +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The YAML format for declaring junctions has not changed, however the way that +multiple junctions interact in a loaded pipeline has changed significantly. + +Specifically, the :ref:`element name ` used to declare +a junction no longer has any special significance, whereas in BuildStream 1 +the junction's name is used to coalesce matching junctions in subprojects. + +BuildStream 2 offers more flexibility in this regard, and allows you to *inherit* +a junction from a subproject, by using a :mod:`link ` element directly +in place of a junction, and/or explicitly override the configuration of a subproject's +junction using the new ``overrides`` configuration attribute which the junction +element now provides. + +Consult the :mod:`junction ` element documentation for a more +detailed explanation. + + +Migrated plugins +---------------- +A majority of the plugins which used to be considered core plugins have been removed +from BuildStream in favor of a more modular and distributed approach. The remaining +core plugins are :ref:`documented here `. + +Any core plugins which you have been using in BuildStream 1 which have been migrated +to separate repositories will need to be accessed externally. + ++---------------+-----------------------------------------------------------------------------------------+ +| Plugin | New location | ++===============+=========================================================================================+ +| **Element plugins** | ++---------------+-----------------------------------------------------------------------------------------+ +| make | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| autotools | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| cmake | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| distutils | `buildstream-plugins `__ (as setuptools) | ++---------------+-----------------------------------------------------------------------------------------+ +| pip | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| meson | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| qmake | `bst-plugins-experimental `_ | ++---------------+-----------------------------------------------------------------------------------------+ +| makemaker | `bst-plugins-experimental `_ | ++---------------+-----------------------------------------------------------------------------------------+ +| modulebuild | `bst-plugins-experimental `_ | ++---------------+-----------------------------------------------------------------------------------------+ +| **Source plugins** | ++---------------+-----------------------------------------------------------------------------------------+ +| bzr | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| git | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| patch | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| pip | `buildstream-plugins `__ | ++---------------+-----------------------------------------------------------------------------------------+ +| deb | `bst-plugins-experimental `_ | ++---------------+-----------------------------------------------------------------------------------------+ +| ostree | `bst-plugins-experimental `_ | ++---------------+-----------------------------------------------------------------------------------------+ +| zip | `bst-plugins-experimental `_ | ++---------------+-----------------------------------------------------------------------------------------+ + +.. attention:: + + **YAML composition with externally loaded plugins** + + Note that when :ref:`YAML composition ` occurs with plugins loaded + from external projects, the *plugin defaults* will be composited with *your project.conf* + and not with the project.conf originating in the external project containing the plugin. + + +Example of externally loaded plugin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +It is recommended to transition directly :ref:`loading these plugins through junctions `, +which can be done as follows. + + +Create an alias for PyPI in your project.conf +''''''''''''''''''''''''''''''''''''''''''''' + +.. code:: yaml + + aliases: + pypi: https://files.pythonhosted.org/packages/ + + +Create buildstream-plugins-junction.bst +''''''''''''''''''''''''''''''''''''''' +Create a junction which accesses the release tarball of the plugin repository. + +.. code:: yaml + + kind: junction + sources: + - kind: tar + url: pypi:e2/d8/ed9e849a1386297f854f9fa0200f3fa108498c0fdb5c86468c1601c7e571/buildstream-plugins-1.91.0.tar.gz + ref: 44c6ea15d15476b68d0767c1d410d416f71544e57be572201058f8b3d3b05f83 + + +Declare the plugin you want to use in your project.conf +''''''''''''''''''''''''''''''''''''''''''''''''''''''' +This will make the ``make`` and ``meson`` element plugins from the +`buildstream-plugins `__ project available for use in your project. + +.. code:: yaml + + plugins: + - origin: junction + junction: buildstream-plugins-junction.bst + elements: + - make + - meson + + +Miscellaneous +------------- +Here we list some miscellaneous breaking changes to the format in general. + + +Element naming +~~~~~~~~~~~~~~ +The names of elements have :ref:`become more restrictive `, for example +they must have the ``.bst`` extension. + + +Overlap whitelist +~~~~~~~~~~~~~~~~~ +The :ref:`overlap whitelist `, which is the public data +found on elements which indicate which files an element can overwrite, must now +be expressed with absolute paths. + + +Strip commands +~~~~~~~~~~~~~~ +The default ``strip-commands`` which :mod:`BuildElement ` implementations +use to split out debug symbols from binaries have been removed. + +This can be solved by declaring a value for the ``%{strip-binaries}`` variable which +will be used for this purpose. diff --git a/_sources/porting_user_configuration.rst.txt b/_sources/porting_user_configuration.rst.txt new file mode 100644 index 000000000..cdf5ff190 --- /dev/null +++ b/_sources/porting_user_configuration.rst.txt @@ -0,0 +1,58 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _porting_user_configuration: + +Porting the buildstream.conf +============================ +This document outlines breaking changes made to the :ref:`user configuration ` +in BuildStream 2. + + +Filename and parallel installation +---------------------------------- +The default filename to load user configuration remains unchanged, however, +if you plan to install and use both versions of BuildStream on the same +host, it is recommended to keep your BuildStream 2 configuration in a +file named ``buildstream2.conf``. + + +Working directories +------------------- +The ``builddir`` and ``artifactdir`` have been removed in favor of the new ``cachedir``. + + +BuildStream 1: +~~~~~~~~~~~~~~ + +.. code:: yaml + + builddir: ${XDG_CACHE_HOME}/buildstream/build + artifactdir: ${XDG_CACHE_HOME}/buildstream/artifacts + + +BuildStream 2: +~~~~~~~~~~~~~~ + +.. code:: yaml + + cachedir: ${XDG_CACHE_HOME}/buildstream/cache + + +Remote cache configuration +-------------------------- +The configuration for remote artifact caches has been completely +redesigned, please refer to the :ref:`artifact cache configuration documentation ` +for details on how to configure remotes in BuildStream 2. diff --git a/_sources/sources/local.rst.txt b/_sources/sources/local.rst.txt new file mode 100644 index 000000000..45a17b108 --- /dev/null +++ b/_sources/sources/local.rst.txt @@ -0,0 +1 @@ +.. automodule:: sources.local diff --git a/_sources/sources/remote.rst.txt b/_sources/sources/remote.rst.txt new file mode 100644 index 000000000..fb0597a70 --- /dev/null +++ b/_sources/sources/remote.rst.txt @@ -0,0 +1 @@ +.. automodule:: sources.remote diff --git a/_sources/sources/tar.rst.txt b/_sources/sources/tar.rst.txt new file mode 100644 index 000000000..983caba55 --- /dev/null +++ b/_sources/sources/tar.rst.txt @@ -0,0 +1 @@ +.. automodule:: sources.tar diff --git a/_sources/sources/workspace.rst.txt b/_sources/sources/workspace.rst.txt new file mode 100644 index 000000000..ebc357245 --- /dev/null +++ b/_sources/sources/workspace.rst.txt @@ -0,0 +1 @@ +.. automodule:: sources.workspace diff --git a/_sources/tutorial/autotools.rst.txt b/_sources/tutorial/autotools.rst.txt new file mode 100644 index 000000000..1907f265f --- /dev/null +++ b/_sources/tutorial/autotools.rst.txt @@ -0,0 +1,162 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _tutorial_autotools: + +Using the autotools element +=========================== +In :ref:`the last chapter ` we observed how the +:mod:`manual ` element works, allowing one to specify and +run commands manually in the process of constructing an *artifact*. + +In this chapter, we'll go over a mostly automated build of a similar +hello world example. We will observe how our configurations of the +:mod:`autotools ` element translate to configurations +on the :mod:`manual ` element, and observe how +:ref:`variable substitution ` works. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/autotools + `_ + subdirectory. + + +Overview +-------- +Instead of using the :mod:`local ` source as we have been using +in the previous examples, we're going to use a :mod:`tar ` source +this time to obtain the ``automake`` release tarball directly from the upstream +hosting. + +In this example we're going to build the example program included in the +upstream ``automake`` tarball itself, and we're going to use the automated +:mod:`autotools ` build element to do so. + + +Project structure +----------------- + + +``project.conf`` +~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/autotools/project.conf + :language: yaml + +Like the :ref:`last project.conf `, we've +added another :ref:`source alias ` for ``gnu``, the location +from which we're going to download the ``automake`` tarball. + + +``elements/base/alpine.bst`` and ``elements/base.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The alpine base and base stack element are defined in the +same way as in the last chapter: :ref:`tutorial_running_commands`. + + +``elements/hello.bst`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/autotools/elements/hello.bst + :language: yaml + +In this case, we haven't touched the element's ``config`` section +at all, instead we just slightly override the bahavior of the +:mod:`autotools ` build element by overriding +the :ref:`command-subdir variable ` + + +Looking at variables +'''''''''''''''''''' +Let's take a moment and observe how :ref:`element composition +` works with variables. + +As :ref:`the documentation ` mentions: + +* The initial settings of the ``project.conf`` variables are setup + using BuildStream's :ref:`builtin defaults `. + +* After this, your local ``project.conf`` may override some variables + on a project wide basis. Those will in turn be overridden by any + defaults provided by element classes, such as the variables set in the + documentation of the :mod:`autotools ` build element. + The variables you set in your final ```` *element declarations*, + will have the final say on the value of a particular variable. + +* Finally, the variables, which may be composed of other variables, + are resolved after all composition has taken place. + +The variable we needed to override was ``command-subdir``, which is an +automatic variable provided by the :mod:`BuildElement ` +abstract class. This variable simply instructs the :mod:`BuildElement ` +in which subdirectory of the ``%{build-root}`` to run its commands in. + +One can always display the resolved set of variables for a given +element's configuration using :ref:`bst show `: + +.. raw:: html + :file: ../sessions/autotools-show-variables.html + +As an exercise, we suggest that you modify the ``hello.bst`` +element to set the prefix like so: + +.. code:: yaml + + variables: + prefix: "/opt" + +And rerun the above :ref:`bst show ` command to observe how this +changes the output. + +Observe where the variables are declared in the :ref:`builtin defaults +` and :mod:`autotools ` element +documentation, and how overriding these effects the resolved set of variables. + + +Using the project +----------------- + + +Build the hello.bst element +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To build the project, run :ref:`bst build ` in the +following way: + +.. raw:: html + :file: ../sessions/autotools-build.html + + +Run the hello world program +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +We probably know by now what's going to happen, but let's run +the program we've compiled anyway using :ref:`bst shell `: + +.. raw:: html + :file: ../sessions/autotools-shell.html + + +Summary +------- +Now we've used an external :ref:`build element `, +from the ``buildstream-plugins`` package and we've taken a look into +:ref:`how variables work `. + +When browsing the :ref:`build elements ` in their +respective documentation, we are now equipped with a good idea of what an element +is going to do, based on their default YAML configuration and any configurations +we have in our project. We can also now observe what variables are in effect +for the build of a given element, using :ref:`bst show `. diff --git a/_sources/tutorial/directives.rst.txt b/_sources/tutorial/directives.rst.txt new file mode 100644 index 000000000..61b4d2cba --- /dev/null +++ b/_sources/tutorial/directives.rst.txt @@ -0,0 +1,188 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _tutorial_directives: + +Optionality and directives +========================== +In this chapter we're going to go over some of the more flexible constructs +which BuildStream offers for :ref:`optionality `, and +show how we can use :ref:`directives ` in the BuildStream +YAML format. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/directives + `_ + subdirectory. + + +Overview +-------- +This chapter's example will build another ``hello.c`` program which much +resembles the program in the :ref:`running commands ` example, +but here we're going to make the greeting string *configurable* using the C +preprocessor. + +We'll be compiling the following C file: + + +``files/src/hello.c`` +~~~~~~~~~~~~~~~~~~~~~ +.. literalinclude:: ../../examples/directives/files/src/hello.c + :language: c + +And we're going to build it using ``make``, using the following Makefile: + + +``files/src/Makefile`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/directives/files/src/Makefile + :language: Makefile + +Notice the addition of ``-DGREETING_MESSAGE="\"${GREETING}\""`` in the above +Makefile, this will allow us to configure the greeting message from the +``hello.bst`` element declaration. + +We will need to add support to our project for *optionality*, and we'll +have to make *conditional statements* to resolve what kind of greeting +we want from the hello world program. + + +Project structure +----------------- +Since this project has much the same structure as the +:ref:`running commands ` chapter did, we won't go over all of +these elements in detail. Instead let's focus on the addition of the new +:ref:`project options ` in ``project.conf``, the added +file in the ``include/`` project subdirectory, and how these come together +in the the ``hello.bst`` element. + + +``project.conf`` +~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/directives/project.conf + :language: yaml + +Here, our ``project.conf`` declares a project option called ``flavor``, and this +will inform what kind of greeting message we want to use when building the project. + + +``elements/hello.bst`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/directives/elements/hello.bst + :language: yaml + +Notice the ``(@)`` symbol we've added in the ``variables:`` section, this +symbol is used to invoke the :ref:`include directive `, +which can be useful for code sharing between elements or simply to improve readability. + +In this case, we are compositing the content of ``include/greeting.bst`` into the +:ref:`variables ` section of the element declaration, directives +can however be used virtually anywhere in the BuildStream YAML format. + + +``include/greeting.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/directives/include/greeting.bst + :language: yaml + +Here we can see the dictionary which will be composited into the ``variables:`` +section of the ``hello.bst`` element described above. + +Note the usage of the ``(?)`` symbol at the toplevel of the YAML dictionary, +this is how we perform :ref:`conditional statements ` +in the BuildStream YAML format. + +This include file uses the ``flavor`` project option we declared in ``project.conf`` to +decide what value will end up being assigned to the ``%{greeting}`` variable, which +will ultimately be used in the ``hello.bst`` element. + + +Using the project +----------------- +Now that we have a project which uses options and conditional statements, +lets build the project with a few different options and observe the outputs. + + +Building hello.bst element with options +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Since the :ref:`flavor option ` we've declared above +has a default, we can build it the first time using :ref:`bst build ` +without any special command line options: + +.. raw:: html + :file: ../sessions/directives-build-normal.html + +If we want to build the ``somber`` flavor, we just need to specify the +additional ``--option`` command line option to :ref:`bst ` +in order to inform BuildStream of the options we want. + +.. raw:: html + :file: ../sessions/directives-build-somber.html + +Note that the ``--option`` option can be specified many times on the +``bst`` command line, so as to support projects which have multiple +options. + +Finally lets get the ``excited`` flavor built as well: + +.. raw:: html + :file: ../sessions/directives-build-excited.html + +If you observe the cache keys above, you will notice that while +we have only three elements in the pipeline, counting ``base/alpine.bst``, +``base.bst`` and ``hello.bst``, we have actually built *five artifacts*, +because the ``hello.bst`` is built differently each time, it has a +different cache key and is stored separately in the artifact cache. + + +Run the hello world program with options +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Since the ``--option`` command line option to :ref:`bst ` +is a main option, it can be used in any command. + +Let's run the ``hello`` program using :ref:`bst shell ` +three times in a row, each time using a different option so we can +observe the results. + + +.. raw:: html + :file: ../sessions/directives-shell-normal.html + + +.. raw:: html + :file: ../sessions/directives-shell-somber.html + + +.. raw:: html + :file: ../sessions/directives-shell-excited.html + + +Summary +------- +In this chapter we've demonstrated how to declare :ref:`project options `, +how to use :ref:`conditional directives `, and also +how to use :ref:`include directives `. + +To get more familliar with these concepts, you may want to explore the remaining +:ref:`directives ` in the BuildStream YAML format, and also take +a look at the various :ref:`types of project options ` that +are also supported. diff --git a/_sources/tutorial/first-project.rst.txt b/_sources/tutorial/first-project.rst.txt new file mode 100644 index 000000000..c5c94079e --- /dev/null +++ b/_sources/tutorial/first-project.rst.txt @@ -0,0 +1,141 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _tutorial_first_project: + +Your first project +================== +To get a feel for the basics, we'll start with the most basic BuildStream project we +could think of. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/first-project + `_ + subdirectory. + + +Creating the project +-------------------- +First, lets create the project itself using the convenience :ref:`bst init ` +command to create a little project structure: + +.. raw:: html + :file: ../sessions/first-project-init.html + + +This will give you a :ref:`project.conf ` which will look like this: + +``project.conf`` +~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/first-project/project.conf + :language: yaml + +The :ref:`project.conf ` is a central point of configuration +for your BuildStream project. + + +Add some content +---------------- +BuildStream processes directory trees as input and output, +so let's just create a ``hello.world`` file for the project +to have. + +.. raw:: html + :file: ../sessions/first-project-touch.html + + +Declare the element +------------------- +Here we're going to declare a simple :mod:`import ` element +which will import the ``hello.world`` file we've created in the previous step. + +Create ``elements/hello.bst`` with the following content: + + +``elements/hello.bst`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/first-project/elements/hello.bst + :language: yaml + + +The source +~~~~~~~~~~ +The :mod:`local ` source used by the ``hello.bst`` element, +can be used to access files or directories which are stored in the same repository +as your BuildStream project. The ``hello.bst`` element uses the :mod:`local ` +source to stage our local ``hello.world`` file. + + +The element +~~~~~~~~~~~ +The :mod:`import ` element can be used to simply add content +directly to the output artifacts. In this case, it simply takes the ``hello.world`` file +provided by its source and stages it directly to the artifact output root. + +.. tip:: + + In this example so far we've used two plugins, the :mod:`local ` + source and the :mod:`import ` element. + + You can always browse the documentation for all plugins in + the :ref:`plugins section ` of the manual. + + +Build the element +----------------- +In order to carry out the activities of the :mod:`import ` element +we've declared, we're going to have to ask BuildStream to *build*. + +This process will collect all of the sources required for the specified ``hello.bst`` +and get the backing :mod:`import ` element to generate an *artifact* +for us. + +.. raw:: html + :file: ../sessions/first-project-build.html + +Now the artifact is ready. + +Using :ref:`bst show `, we can observe that the artifact's state, which was reported +as ``buildable`` in the :ref:`bst build ` command above, has now changed to ``cached``: + +.. raw:: html + :file: ../sessions/first-project-show.html + + +Observe the output +------------------ +Now that we've finished building, we can checkout the output of the +artifact we've created using :ref:`bst artifact checkout ` + +.. raw:: html + :file: ../sessions/first-project-checkout.html + +And observe that the file we expect is there: + +.. raw:: html + :file: ../sessions/first-project-ls.html + + +Summary +------- +In this section we've created our first BuildStream project from +scratch, but it doesnt do much. + +We've observed the general structure of a BuildStream project, +and we've run our first build. diff --git a/_sources/tutorial/integration-commands.rst.txt b/_sources/tutorial/integration-commands.rst.txt new file mode 100644 index 000000000..16fd69aa5 --- /dev/null +++ b/_sources/tutorial/integration-commands.rst.txt @@ -0,0 +1,145 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _tutorial_integration_commands: + +Integration commands +==================== +Sometimes a software requires more configuration or processing than what is +performed at installation time, otherwise it will not run properly. + +This is especially true in cases where a daemon or library interoperates +with third party extensions and needs to maintain a system wide cache whenever +its extensions are installed or removed; system wide font caches are an example +of this. + +In these cases we use :ref:`integration commands ` to +ensure that a runtime is ready to run after all of its components have been *staged*. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/integration-commands + `_ + subdirectory. + + +Overview +-------- +In this chapter, we'll be exploring :ref:`integration commands `, +which will be our first look at :ref:`public data `. + + +Project structure +----------------- + + +``project.conf`` and ``elements/base.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The project.conf and base stack :mod:`stack ` element are configured in the +same way as in the previous chapter: :ref:`tutorial_running_commands`. + + +``elements/base/alpine.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/integration-commands/elements/base/alpine.bst + :language: yaml + +This is the same ``base/alpine.bst`` we've seen in previous chapters, +except that we've added an :ref:`integration command `. + +This informs BuildStream that whenever the output of this element is +expected to *run*, this command should be run first. In this case we +are simply running ``ldconfig`` as a precautionary measure, to ensure +that the runtime linker is ready to find any shared libraries we may +have added to ``%{libdir}``. + + +Looking at public data +'''''''''''''''''''''' +The :ref:`integration commands ` used here is the first time +we've used any :ref:`builtin public data `. + +Public data is a free form portion of an element's configuration and +is not necessarily understood by the element on which it is declared, public +data is intended to be read by its reverse dependency elements. + +This allows annotations on some elements to inform elements later in +the dependency chain about details of its artifact, or to suggest how +it should be processed. + + +``elements/libhello.bst`` and ``elements/hello.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +These are basically manual elements very similar to the ones we've +seen in the previous chapter: :ref:`tutorial_running_commands`. + +These produce a library and a hello program which uses the library, +we will consider these irrelevant to the topic and leave examination +of `their sources +`_ +as an exercise for the reader. + + +Using the project +----------------- + + +Build the hello.bst element +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To build the project, run :ref:`bst build ` in the +following way: + +.. raw:: html + :file: ../sessions/integration-commands-build.html + +Observe in the build process above, the integration command declared on the +``base/alpine.bst`` element is run after staging the dependency artifacts +into the build sandbox and before running any of the build commands, for +both of the ``libhello.bst`` and ``hello.bst`` elements. + +BuildStream assumes that commands which are to be run in the build sandbox +need to be run in an *integrated* sandbox. + +.. tip:: + + Integration commands can be taxing on your overall build process, + because they need to run at the beginning of every build which + :ref:`runtime depends ` on the element + declaring them. + + For this reason, it is better to leave out more onerous tasks + if they are not needed at software build time, and handle those + specific tasks differently later in the pipeline, before deployment. + + +Run the hello world program +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Unlike the previous chapters, this hello world program takes an argument, +we can invoke the program using :ref:`bst shell `: + +.. raw:: html + :file: ../sessions/integration-commands-shell.html + +Here we see again, the integration commands are also used when preparing +the shell to launch a command. + + +Summary +------- +In this chapter we've observed how :ref:`integration commands ` +work, and we now know about :ref:`public data `, which plugins +can read from their dependencies in order to influence their build process. diff --git a/_sources/tutorial/running-commands.rst.txt b/_sources/tutorial/running-commands.rst.txt new file mode 100644 index 000000000..8be85dc16 --- /dev/null +++ b/_sources/tutorial/running-commands.rst.txt @@ -0,0 +1,249 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _tutorial_running_commands: + +Running commands +================ +In :ref:`the first chapter ` we only imported +a file to create an artifact, this time lets run some commands inside +the :ref:`isolated build sandbox `. + +.. note:: + + This example is distributed with BuildStream + in the `doc/examples/running-commands + `_ + subdirectory. + + +Overview +-------- +In this chapter, we'll be running commands inside the sandboxed +execution environment and producing build output. + +We'll be compiling the following simple C file: + + +``files/src/hello.c`` +~~~~~~~~~~~~~~~~~~~~~ +.. literalinclude:: ../../examples/running-commands/files/src/hello.c + :language: c + + +And we're going to build it using ``make``, using the following Makefile: + + +``files/src/Makefile`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/running-commands/files/src/Makefile + :language: Makefile + + +We'll be using the most fundamental :ref:`build element `, +the :mod:`manual ` build element. + +The :mod:`manual ` element is the backbone on which all the other +build elements are built, so understanding how it works at this level is helpful. + + +Project structure +----------------- +In this project we have a ``project.conf``, a directory with some source +code, and 3 element declarations. + +Let's first take a peek at what we need to build using :ref:`bst show `: + +.. raw:: html + :file: ../sessions/running-commands-show-before.html + +This time we have loaded a pipeline with 3 elements, let's go over what they do +in detail. + + +.. _tutorial_running_commands_project_conf: + +``project.conf`` +~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/running-commands/project.conf + :language: yaml + +Our ``project.conf`` is very much like the last one, except that we +have defined a :ref:`source alias ` for ``alpine``. + +.. tip:: + + Using :ref:`source aliases ` for groups of sources + which are generally hosted together is encouraged. This allows one to globally + change the access scheme or URL for a group of repositories which belong together. + + +``elements/base/alpine.bst`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/running-commands/elements/base/alpine.bst + :language: yaml + +This :mod:`import ` element uses a :mod:`tar ` +source to download our Alpine Linux tarball to create our base runtime. + +This tarball is a sysroot which provides the C runtime libraries +and some programs - this is what will be providing the programs we're +going to run in this example. + + +``elements/base.bst`` +~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/running-commands/elements/base.bst + :language: yaml + +This is just a symbolic :mod:`stack ` element which declares that +anything which depends on it, will implicitly depend on ``base/alpine.bst``. + +It is typical to use stack elements in places where the implementing logical +software stack could change, but you rather not have your higher level components +carry knowledge about those changing components. + +Any element which :ref:`runtime depends ` on +the ``base.bst`` will now be able to execute programs provided by the imported +``base/alpine.bst`` runtime. + + +``elements/hello.bst`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. literalinclude:: ../../examples/running-commands/elements/hello.bst + :language: yaml + +Finally we have the element which executes commands. Looking at the +:mod:`manual ` element's documentation, we can see that +the element configuration exposes four command lists: + +* ``configure-commands`` + + Commands which are run in preparation of a build. This is where you + would normally call any configure stage build tools to configure + the build how you like and generate some files needed for the build. + +* ``build-commands`` + + Commands to run the build, usually a build system will + invoke the compiler for you here. + +* ``install-commands`` + + Commands to install the build results. + + Commands to install the build results into the target system, + these should install files somewhere under ``%{install-root}``. + +* ``strip-commands`` + + Commands to doctor the build results after the install. + + Typically this involves stripping binaries of debugging + symbols or stripping timestamps from build results to ensure + reproducibility. + +.. tip:: + + All other :ref:`build elements ` + implement exactly the same command lists too, except that they provide + default commands specific to invoke the build systems they support. + +The :mod:`manual ` element however is the most basic +and does not provide any default commands, so we have instructed it +to use ``make`` to build and install our program. + + +Using the project +----------------- + + +Build the hello.bst element +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To build the project, run :ref:`bst build ` in the +following way: + +.. raw:: html + :file: ../sessions/running-commands-build.html + +Now we've built our hello world program, using ``make`` +and the C compiler provided by the Alpine Linux image. + +In the :ref:`first chapter ` we observed that the inputs +and output of an element are *directory trees*. In this example, the directory tree +generated by ``base/alpine.bst`` is consumed by ``hello.bst`` due to the +:ref:`implicit runtime dependency ` introduced by ``base.bst``. + +.. tip:: + + All of the :ref:`dependencies ` which are required to run for + the sake of a build, are staged at the root of the build sandbox. These comprise the + runtime environment in which the depending element will run commands. + + The result is that the ``make`` program and C compiler provided by ``base/alpine.bst`` + were already in ``$PATH`` and ready to run when the commands were needed by ``hello.bst``. + +Now observe that all of the elements in the loaded pipeline are ``cached``, +the element is *built*: + +.. raw:: html + :file: ../sessions/running-commands-show-after.html + + +Run the hello world program +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Now that we've built everything, we can indulge ourselves in running +the hello world program using :ref:`bst shell `: + +.. raw:: html + :file: ../sessions/running-commands-shell.html + +Here, :ref:`bst shell ` created a runtime environment for running +the ``hello.bst`` element. This was done by staging all of the dependencies of +``hello.bst`` including the ``hello.bst`` output itself into a directory. Once a directory +with all of the dependencies was staged and ready, we ran the ``hello`` command from +within the build sandbox environment. + +.. tip:: + + When specifying a command for :ref:`bst shell ` to run, + we always specify ``--`` first. This is a commonly understood shell syntax + to indicate that the remaining arguments are to be treated literally. + + Specifying ``--`` is optional and disambiguates BuildStream's arguments + and options from those of the program being run by + :ref:`bst shell `. + + +Summary +------- +In this chapter we've explored how to use the :mod:`manual ` element, +which forms the basis of all build elements. + +We've also observed how the directory tree from the output *artifact* of one element +is later *staged* at the root of the sandbox, as input for use by any build elements +which :ref:`depend ` on that element. + +.. tip:: + + The way that elements consume their dependency input can vary across the + different *kinds* of elements. This chapter describes how it works for + :mod:`build elements ` implementations, which + are the most commonly used element type. diff --git a/_sources/using_commands.rst.txt b/_sources/using_commands.rst.txt new file mode 100644 index 000000000..aca308e9d --- /dev/null +++ b/_sources/using_commands.rst.txt @@ -0,0 +1,296 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +.. _commands: + +Commands +======== +This page contains documentation for each BuildStream command, +along with their possible options and arguments. Each command can be +invoked on the command line, where, in most cases, this will be from the +project's main directory. + + +Commonly used parameters +------------------------ + + +.. _invoking_specify_remotes: + +Remotes +~~~~~~~ +Remote :ref:`cache servers ` can be specified on the +command line for commands which may result in communicating with such servers. + +Any command which has arguments to specify a ``REMOTE``, such as ``--artifact-remote`` +or ``--source-remote``, will override whatever was set in the user configuration, +and will have an accompanying switch which allows the command to decide whether +to ignore any remote :ref:`artifact ` or :ref:`source ` +caches suggested by project configuration. + +Remotes can be specified on the command line either as a simple URI, or as +a comma separated list of key value pairs. + +**Specifying a remote using a URI** + +.. code:: shell + + bst artifact push --remote https://artifacts.com/artifacts:8088 element.bst + +**Specifying a remote using key value pairs** + +.. code:: shell + + bst build --artifact-remote \ + url=https://artifacts.com/artifacts:8088,type=index,server-cert=~/artifacts.cert \ + element.bst + + +Attributes +'''''''''' +Here is the list attributes which can be spefied when providing a ``REMOTE`` on the command line: + +* ``url`` + + The URL of the remote, possibly including a port number. + +* ``instance-name`` + + The instance name of this remote, used for sharding by some implementations. + +* ``type`` + + Whether this remote is to be used for indexing, storage or both, as explained + in the corresponding :ref:`user configuration documentation ` + +* ``push`` + + Normally one need not specify this, as it is often inferred by the command + being used. In some cases, like :ref:`bst build `, it can + be useful to specify multiple remotes, and only allow pushing to some of + the remotes. + + If unspecified, this is assumed to be ``True`` and BuildStream will attempt + to push to the remote, but fallback to only pulling if insufficient credentials + were provided. + +* ``server-cert``, ``client-cert``, ``client-key``: + + These keys specify the attributes of the :ref:`authentication configuration `. + + When specifying these on the command line, they are interpreted as paths relative + to the current working directory. + + +Top-level commands +------------------ + +.. The bst options e.g. bst --version, or bst --verbose etc. +.. _invoking_bst: + +.. click:: buildstream._frontend:cli + :prog: bst + +.. Further description of the command goes here + +---- + +.. _invoking_artifact: + +.. click:: buildstream._frontend.cli:artifact + :prog: bst artifact + +---- + +.. the `bst init` command +.. _invoking_init: + +.. click:: buildstream._frontend.cli:init + :prog: bst init + +---- + +.. the `bst build` command +.. _invoking_build: + +.. click:: buildstream._frontend.cli:build + :prog: bst build + +---- + +.. _invoking_show: + +.. click:: buildstream._frontend.cli:show + :prog: bst show + +---- + +.. _invoking_shell: + +.. click:: buildstream._frontend.cli:shell + :prog: bst shell + +---- + +.. _invoking_source: + +.. click:: buildstream._frontend.cli:source + :prog: bst source + +---- + +.. _invoking_workspace: + +.. click:: buildstream._frontend.cli:workspace + :prog: bst workspace + + +.. _artifact_subcommands: + +Artifact subcommands +-------------------- + + +.. _artifact_names: + +Artifact names +~~~~~~~~~~~~~~ +Various artifact subcommands accept either :ref:`element names `, +which will operate on artifacts by deriving the artifact from local project state, +or :term:`artifact names ` interchangeably as targets. Artifact names allow +the user to operate directly on cached artifacts, without requiring local project data. + +An artifact name is composed of the following identifiers: + +* The :ref:`project name ` + +* The :ref:`element name `, without any trailing ``.bst`` extension + +* The cache key of the element at the time it was built. + +To compose an artifact name, simply join these using a forward slash (``/``) character, like so: ``//``. + +An artifact name might look like: ``project/target/788da21e7c1b5818b7e7b60f7eb75841057ff7e45d362cc223336c606fe47f27`` + + +.. _invoking_artifact_checkout: + +.. click:: buildstream._frontend.cli:artifact_checkout + :prog: bst artifact checkout + +---- + +.. _invoking_artifact_log: + +.. click:: buildstream._frontend.cli:artifact_log + :prog: bst artifact log + +---- + +.. _invoking_artifact_pull: + +.. click:: buildstream._frontend.cli:artifact_pull + :prog: bst artifact pull + +---- + +.. _invoking_artifact_push: + +.. click:: buildstream._frontend.cli:artifact_push + :prog: bst artifact push + +---- + +.. _invoking_artifact_delete: + +.. click:: buildstream._frontend.cli:artifact_delete + :prog: bst artifact delete + +---- + +.. _invoking_artifact_show: + +.. click:: buildstream._frontend.cli:artifact_show + :prog: bst artifact show + +---- + +.. _invoking_artifact_list_contents: + +.. click:: buildstream._frontend.cli:artifact_list_contents + :prog: bst artifact list-contents + + +.. _source_subcommands: + +Source subcommands +------------------ + +.. _invoking_source_fetch: + +.. click:: buildstream._frontend.cli:source_fetch + :prog: bst source fetch + +---- + +.. _invoking_source_track: + +.. click:: buildstream._frontend.cli:source_track + :prog: bst source track + +---- + +.. _invoking_source_push: + +.. click:: buildstream._frontend.cli:source_push + :prog: bst source push + +---- + +.. _invoking_source_checkout: + +.. click:: buildstream._frontend.cli:source_checkout + :prog: bst source checkout + + +.. _workspace_subcommands: + +Workspace subcommands +--------------------- + +.. _invoking_workspace_open: + +.. click:: buildstream._frontend.cli:workspace_open + :prog: bst workspace open + +---- + +.. _invoking_workspace_close: + +.. click:: buildstream._frontend.cli:workspace_close + :prog: bst workspace close + +---- + +.. _invoking_workspace_reset: + +.. click:: buildstream._frontend.cli:workspace_reset + :prog: bst workspace reset + +---- + +.. _invoking_workspace_list: + +.. click:: buildstream._frontend.cli:workspace_list + :prog: bst workspace list diff --git a/_sources/using_config.rst.txt b/_sources/using_config.rst.txt new file mode 100644 index 000000000..0bc1ba62e --- /dev/null +++ b/_sources/using_config.rst.txt @@ -0,0 +1,1318 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +.. _user_config: + + +User configuration +================== +User configuration and preferences can be specified in a user provided +configuration file, and in most cases these preferences can be overridden +using :ref:`command line options `. + +Values that are not specified in a user configuration file assume +their :ref:`default values `. + + +Configuration file +------------------ +Unless a configuration file is explicitly specified on the :ref:`command line ` +when invoking ``bst``, an attempt is made to load user specific configuration from +``$XDG_CONFIG_HOME/buildstream.conf``. On most Linux based systems, the location +will be ``~/.config/buildstream.conf`` + +.. note:: + + If you have have multiple major versions of BuildStream installed, you + can have separate configuration files in your ``${XDG_CONFIG_HOME}``. + + You can do this by naming them according to the major versions of + BuildStream you have installed. BuildStream 1 will load its configuration + from ``$XDG_CONFIG_HOME/buildstream1.conf`` and BuildStream 2 will load + its configuration from ``$XDG_CONFIG_HOME/buildstream2.conf``, while + any version will fallback to ``$XDG_CONFIG_HOME/buildstream.conf``. + + + +Working directories +------------------- +The working directories control where BuildStream will store data. While +these will have sane defaults, you may want to change these directories +depending on your partitioning scheme and where you may have extra space. + +Environment variables and ``~/`` home directory expressions are supported +when specifying working directories. + +Working directories are configured at the toplevel of your configuration file, like so: + +.. code:: yaml + + # + # Configure working directories + # + sourcedir: ~/buildstream/sources + + +Attributes +~~~~~~~~~~ + +* ``sourcedir`` + + This is the location where BuildStream stores the source code it downloads + for builds. + +* ``logdir`` + + This is the location where BuildStream stores log files of build command + output and basically logs pertaining to any activity BuildStream orchestrates. + +* ``cachedir`` + + This is the location where BuildStream stores the local *CAS* (*Content Addressable Storage*). + + The *CAS* is used to cache anything and everything which BuildStream may + reuse at a later time. + + .. attention:: + + While it may be beneficial at times to delete the entire cache directory + manually in order to free up disk space, one should keep in mind that + the ``sourcedir`` and ``logdir`` are configured as subdirectories of + this directory when default configuration is used. + + Take care not to accidentally remove all your cached downloaded sources + when deleting your cache. + +* ``workspacedir`` + + A default location for :ref:`opening workspaces `. + + .. note:: + + By default this is configured to ``.``, which is to say workspaces are + created as a subdirectory of the current working directory by default. + + Because of this, the ``workspacedir`` directory is the only directory + which is allowed to be declared as a relative path. + + +.. _config_local_cache: + +Cache control +------------- +Beyond deciding what directory you intend to place the cache, there are +some controls on what is cached locally and how. + +These are controlled by the attributes of a ``cache`` dictionary at the +toplevel of your configuration file, like so: + +.. code:: + + # + # Control the cache + # + cache: + + # Use as much space as possible + quota: infinity + + # Keep 5% of disk space available + reserved-disk-space: 5% + + # Retain 80% of the cache on cleanup + low-watermark: 80% + + # Avoid pulling large amounts of data we don't need locally + pull-buildtrees: False + + # + # Avoid caching build trees if we don't need them + cache-buildtrees: auto + + # Connection config is parameters given to grpc. It's completely + # optional. By default keepalive time is unset and grpc defaults + # are used. + connection-config: + keepalive-time: 60 + + # + # Support CAS server as remote cache + # Useful to minimize network traffic with remote execution + # or to work with limited local disk space + storage-service: + url: https://cache-server.com/cas:11001 + auth: + server-cert: server.crt + client-cert: client.crt + client-key: client.key + access-token: access.token + access-token-reload-interval: 60 + + +Attributes +~~~~~~~~~~ + +* ``quota`` + + This controls how much data you allow BuildStream to cache locally. + + An attempt will be made to error out instead of exceeding the maximum + quota which the user has allowed here. Given that it is impossible for + BuildStream to know how much data a given build will create, this quota + is implemented on a best effort basis. + + The ``quota`` can be specified in multiple ways: + + * The special ``infinity`` value + + This default value states that BuildStream can use as much space as + is available on the filesystem where the cache resides. + + * A number in bytes. + + * A human readable number, suffixed in K, M, G or T + + E.g. ``250K`` being 250 kilobytes, ``100M`` being 100 megabytes, etc. + + * A percentage value, e.g. ``80%`` + + Percentage values are taken to represent a percentage of the partition + size on the filesystem where the cache has been configured. + +* ``reserved-disk-space`` + + This controls how much disk space should remain available. If the amount + of available disk space falls below the specified value, unused cache + objects will be pruned even if the configured quota has not been exceeded. + + ``reserved-disk-space`` can be specified in the same way as ``quota``, with + the exception of the special ``infinity`` value. The default is ``5%``. + +* ``low-watermark`` + + This controls how much of the cache should be retained on cleanup. + + ``low-watermark`` is specified as a percentage of the effective cache quota + as configured by ``quota`` and/or ``reserved-disk-space``. The default is + ``80%``, which means that when cleanup is triggered, 20% of the cache will + be pruned by removing CAS objects that haven't been used recently. + +* ``pull-buildtrees`` + + Whether to pull *build trees* when downloading remote artifacts. + + The *build tree* of an artifact is the directory where a build took + place, this is useful for :ref:`running a build shell ` + in order to observe how an element was built or to debug how a + build failed if the build failed remotely. + + Since build trees are rather expensive, the default is to not pull + build trees for every artifact. If you need a build tree that exists + remotely, it will be possible to download it as an option at the + time you run a command which requires it. + +* ``cache-buildtrees`` + + Whether to cache build trees when creating artifacts, if build trees + are cached locally and the client is configured to push to remote servers, + then build trees will be pushed along with any uploaded artifacts. + + This configuration has three possible values: + + * ``never``: Never cache build trees + * ``auto``: Only cache the build trees where necessary (e.g. for failed builds) + * ``always``: Always cache the build tree. + +* ``storage-service`` + + An optional :ref:`service configuration ` + to use a *Content Addressable Storage* service as a remote cache. Write access + is required. + + This service is compatible with the *storage* service offered by + :ref:`cache servers `. + + Without this option, all content is stored in the local cache. This includes + CAS objects from fetched sources, build outputs and pulled artifacts. + With this option, content is primarily stored in the remote cache and the + local cache is populated only as needed. E.g. ``bst artifact checkout`` + will download CAS objects on demand from the remote cache. + This feature is incompatible with offline operation. + + This is primarily useful in combination with + :ref:`remote execution ` to minimize downloads + of build outputs, which may not be needed locally. The elimination of + unnecessary downloads reduces total build time, especially if the bandwidth + between the local system and the remote execution cluster is limited. + + .. tip:: + + Skip the ``storage-service`` option in the + :ref:`remote execution ` configuration to + use the same CAS service for caching and remote execution. + + It is also possible to configure this with local builds without remote + execution. This enables operation with a small local cache even with large + projects. However, for local builds this can result in a longer total build + time due to additional network transfers. This is only recommended with a + high bandwidth connection to a storage-service, ideally in a local network. + + +Scheduler controls +------------------ +Controls related to how the scheduler works are exposed as attributes of the +toplevel ``scheduler`` dictionary, like so: + +.. code:: yaml + + # + # Control the scheduler + # + scheduler: + + # Allow building up to four seperate elements at a time + builders: 4 + + # Continue building as many elements as possible if anything fails + on-error: continue + + +Attributes +~~~~~~~~~~ + +* ``fetchers`` + + The number of concurrent tasks which download sources or artifacts. + +* ``pushers`` + + The number of concurrent tasks which upload sources or artifacts. + +* ``builders`` + + The number of concurrent tasks which build elements. + + .. note:: + + This does not control the number of processes in the scope of the + build of a single element, but rather the number of elements which + may be built in parallel. + +* ``network-retries`` + + The number of times to retry a task which failed due to network connectivity issues. + +* ``on-error`` + + What to do when a task fails and BuildStream is running in non-interactive mode. This can + be set to the following values: + + * ``continue``: Continue with other tasks, a summary of errors will be printed at the end + * ``quit``: Quit after all ongoing tasks have completed + * ``terminate``: Abort any ongoing tasks and exit immediately + + .. note:: + + If BuildStream is running in interactive mode, then the ongoing build will be suspended + and the user will be prompted and asked what to do when a task fails. + + Interactive mode is automatically enabled if BuildStream is connected to a terminal + rather than being run automatically, or, it can be specified on the :ref:`command line `. + + +Build controls +-------------- +Some aspects about how elements get built can be controlled by attributes of the ``build`` +dictionary at the toplevel, like so: + +.. code:: yaml + + # + # Build controls + # + build: + + # + # Allow up to 4 parallel processes to execute within the scope of one build + # + max-jobs: 4 + + +Attributes +~~~~~~~~~~ + +* ``max-jobs`` + + This is a best effort attempt to instruct build systems on how many parallel + processes to use when building an element. + + It is supported by most popular build systems such as ``make``, ``cmake``, ``ninja``, + etc, via environment variables such as ``MAXJOBS`` and similar command line options. + + When using the special value ``0``, BuildStream will allocate the number of threads + available on the host and limit this with a hard coded value of ``8``, which was + found to be an optimial number when building even on hosts with many cores. + +* ``retry-failed`` + + Try to build elements for which a failed build artifact is found when running + :ref:`bst build `. + + This is useful in the case that a build has failed due to insufficient resources + such as memory or disk space. + +* ``dependencies`` + + This instructs what dependencies of the target elements should be built, valid + values for this attribute are: + + * ``none``: Only build elements required to generate the expected target artifacts + * ``all``: Build elements even if they are build dependencies of artifacts which are already cached + + +Fetch controls +-------------- +Some aspects about how sources get fetched can be controlled by attributes of the ``fetch`` +dictionary at the toplevel, like so: + +.. code:: yaml + + # + # Fetch controls + # + fetch: + + # + # Don't allow fetching from project defined alias or mirror URIs + # + source: user + + +Attributes +~~~~~~~~~~ + +* ``source`` + + This controls what URIs are allowed to be accessed when fetching sources, + valid values for this attribute are: + + * ``all``: Fetch from mirrors defined in :ref:`user configuration ` and + :ref:`project configuration `, and also project defined + :ref:`default alias URIs `. + * ``aliases``: Only allow fetching from project defined :ref:`default alias URIs `. + * ``mirrors``: Only allow fetching from mirrors defined in :ref:`user configuration ` and + :ref:`project configuration ` + * ``user``: Only allow fetching from mirrors defined in :ref:`user configuration ` + + +Track controls +-------------- +Some aspects about how sources get tracked can be controlled by attributes of the ``track`` +dictionary at the toplevel, like so: + +.. code:: yaml + + # + # Track controls + # + track: + + # + # Only track sources for new refs from project defined default alias URIs + # + source: aliases + + +Attributes +~~~~~~~~~~ + +* ``source`` + + This controls what URIs are allowed to be accessed when tracking sources + for new refs, valid values for this attribute are: + + * ``all``: Track from mirrors defined in :ref:`user configuration ` and + :ref:`project configuration `, and also project defined + :ref:`default alias URIs `. + * ``aliases``: Only allow tracking from project defined :ref:`default alias URIs `. + * ``mirrors``: Only allow tracking from mirrors defined in :ref:`user configuration ` and + :ref:`project configuration ` + * ``user``: Only allow tracking from mirrors defined in :ref:`user configuration ` + + +Logging controls +---------------- +Various aspects of how BuildStream presents output and UI can be controlled with +attributes of the toplevel ``logging`` dictionary, like so: + +.. code:: yaml + + # + # Control logging output + # + logging: + + # + # Whether to be verbose + # + verbose: True + + +Attributes +~~~~~~~~~~ + +* ``verbose`` + + Whether to use verbose logging. + +* ``debug`` + + Whether to print messages related to debugging BuildStream itself. + +* ``key-length`` + + When displaying abbreviated cache keys, this controls the number of characters + of the cache key which should be printed. + +* ``throttle-ui-updates`` + + Whether the throttle updates to the status bar in interactive mode. If set to ``True``, + then the status bar will be updated once per second. + +* ``error-lines`` + + The maximum number of lines to print in the main logging output related to an + error processing an element, these will be the last lines found in the relevant + element's stdout and stderr. + +* ``message-lines`` + + The maximum number of lines to print in a detailed message sent to the main logging output. + +* ``element-format`` + + The default format to use when printing out elements in :ref:`bst show ` + output, and when printing the pipeline summary at the beginning of sessions. + + The format is specified as a string containing variables which will be expanded + in the resulting string, variables must be specified using a leading percent sign + and enclosed in curly braces, a colon can be specified in the variable to perform + python style string alignments, e.g.: + + .. code:: yaml + + logging: + + # + # Set the element format + # + element-format: | + + %{state: >12} %{full-key} %{name} %{workspace-dirs} + + Variable names which can be used in the element format consist of: + + * ``name`` + + The :ref:`element path `, which is the name of the element including + any leading junctions. + + * ``key`` + + The abbreviated cache key, the length of which is controlled by the ``key-length`` logging configuration. + + * ``full-key`` + + The full cache key. + + * ``state`` + + The element state, this will be formatted as one of the following: + + * ``no reference``: If the element still needs to be :ref:`tracked `. + * ``junction``: If the element is a junction and as such does not have any relevant state. + * ``failed``: If the element has been built and the build has failed. + * ``cached``: If the element has been successfully built and is present in the local cache. + * ``fetch needed``: If the element cannot be built yet because the sources need to be :ref:`fetched `. + * ``buildable``: If the element has all of its sources and build dependency artifacts cached locally. + * ``waiting``: If the element has all of its sources cached but its build dependencies are not yet locally cached. + + * ``config`` + + The :ref:`element configuration `, formatted in YAML. + + * ``vars`` + + The resolved :ref:`element variables `, formatted as a simple YAML dictionary. + + * ``env`` + + The resolved :ref:`environment variables `, formatted as a simple YAML dictionary. + + * ``public`` + + The resolved :ref:`public data `, formatted in YAML. + + * ``workspaced`` + + If the element has an open workspace, this will expand to the string *"(workspaced)"*, otherwise + it will expand to an empty string. + + * ``workspace-dirs`` + + If the element has an open workspace, this will expand to the workspace directory, prefixed with + the text *"Workspace: "*, otherwise it will expand to an empty string. + + * ``deps`` + + A list of the :ref:`element paths ` of all dependency elements. + + * ``build-deps`` + + A list of the :ref:`element paths ` of all build dependency elements. + + * ``runtime-deps`` + + A list of the :ref:`element paths ` of all runtime dependency elements. + +* ``message-format`` + + The format to use for messages being logged in the aggregated main logging output. + + Similarly to the ``element-format``, The format is specified as a string containing variables which + will be expanded in the resulting string, and variables must be specified using a leading percent sign + and enclosed in curly braces, e.g.: + + .. code:: yaml + + logging: + + # + # Set the message format + # + message-format: | + + [%{elapsed}][%{key}][%{element}] %{action} %{message} + + Variable names which can be used in the element format consist of: + + * ``elapsed`` + + If this message announces the completion of (successful or otherwise) of an activity, then + this will expand to a time code showing how much time elapsed for the given activity, in + the format: ``HH:MM:SS``, otherwise an empty time code will be displayed in the format: + ``--:--:--``. + + * ``elapsed-us`` + + Similar to the ``elapsed`` variable, however the timecode will include microsecond precision. + + * ``wallclock`` + + This will display a timecode for each message displaying the local wallclock time, in the + format ``HH:MM:SS``. + + * ``wallclock-us`` + + Similar to the ``wallclock`` variable, however the timecode will include microsecond precision. + + * ``key`` + + The abbreviated cache key of the element the message is related to, the length of which is controlled + by the ``key-length`` logging configuration. + + If the message in question is not related to any element, then this will expand to whitespace + of equal length. + + * ``element`` + + This will be formatted to an indicator consisting of the type of activity which is being + performed on the element (e.g. *"build"* or *"fetch"* etc), and the :ref:`element path ` + of the element this message is related to. + + If the message in question is not related to any element, then a string will be formatted + to indicate that this message is related to a core activity instead. + + * ``action`` + + A classifier of the type of message this is, the possible values this will expand to are: + + * ``DEBUG`` + + This is a message related to debugging BuildStream itself + + * ``STATUS`` + + A message showing some detail of what is currently happening, this message will not + be displayed unless verbose output is enabled. + + * ``INFO`` + + An informative message, this may be printed for instance when discovering a new + ref for source code when running :ref:`bst source track `. + + * ``WARN`` + + A warning message. + + * ``ERROR`` + + An error message. + + * ``BUG`` + + A bug happened in BuildStream, this will usually be accompanied by a python stack trace. + + * ``START`` + + An activity related to an element started. + + Any ``START`` message will always be accompanied by a later ``SUCCESS``, ``FAILURE`` + or ``SKIPPED`` message. + + * ``SUCCESS`` + + An activity related to an element completed successfully. + + * ``FAILURE`` + + An activity related to an element failed. + + * ``SKIPPED`` + + After strating this activity, it was discovered that no work was needed and + the activity was skipped. + + * ``message`` + + The brief message, or the path to the corresponding log file, will be printed here. + + When this is a scheduler related message about the commencement or completion of + an element related activity, then the path to the corresponding log for that activity + will be printed here. + + If it is a message issued for any other reason, then the message text will be formatted here. + + .. note:: + + Messages issued by the core or by plugins are allowed to provide detailed accounts, these + are the indented multiline messages which sometimes get displayed in the main aggregated + logging output, and will be printed regardless of the logging ``message-format`` value. + + +Remote services +--------------- +BuildStream can be configured to cooperate with remote caches and +execution services. + + +.. _config_remote_auth: + +Authentication +~~~~~~~~~~~~~~ +BuildStream supports end to end encryption when communicating with remote +services. + +All remote service configuration blocks come with an optional ``auth`` +configuration block which allows one to specify the certificates +and keys required for encrypted traffic. + +The ``auth`` configuration block looks like this: + +.. code:: yaml + + auth: + server-cert: server.crt + client-cert: client.crt + client-key: client.key + access-token: access.token + access-token-reload-interval: 60 + + +Attributes +'''''''''' + +* ``server-cert`` + + The server certificate is used to verify the identity of the server instead + of using the CA store from the operating system for verification. + +* ``client-cert`` + + The client certificate is used by the remote server to authenticate the client. + +* ``client-key`` + + The private client key corresponding to the specified client certificate. + +* ``access-token`` + + The path to a token for optional HTTP bearer authentication. + +* ``access-token-reload-interval`` + + The reload interval in minutes for the specified access token. + If not specified, automatic reloading is disabled. + +Remote cache services may allow *downloading* artifacts and sources without +authentication, in which case only ``server-cert`` is required for secure access +(or no attributes at all if the CA store from the OS can be used). + +However, remote cache services should normally not allow *uploading* artifacts +or sources without authentication. Authentication can be configured by setting +``access-token`` or both ``client-key`` and ``client-cert``. + + +.. _config_cache_servers: + +Cache servers +~~~~~~~~~~~~~ +BuildStream supports two types of cache servers, :ref:`source cache servers ` +and :ref:`artifact cache servers `. These services allow you +to store sources and build artifacts for later reuse, and share them among your +peers. + +.. important:: + + **Storing and indexing** + + Cache servers are split into two separate services, the *index* and the *storage*. + Sometimes these services are provided by the same server, and sometimes it is desirable + to use different cache servers for indexing and storing data. + + In simple setups, it is possible to use the same cache server for indexing and storing + of both sources and artifacts. However, when using :ref:`remote execution ` + it is recommended to use the remote execution build cluster's ``storage-service`` as the *storage* + service of your cache servers, which may require setting up your *index* service separately. + + When configuring cache servers, BuildStream will require both storage and indexing capabilities, + otherwise no attempt will be made to fetch or push data to and from cache servers. + +Cache server configuration is declared in the following way: + +.. code:: yaml + + override-project-caches: false + servers: + - url: https://cache-server.com/cache:11001 + instance-name: main + type: all + push: true + auth: + server-cert: server.crt + client-cert: client.crt + client-key: client.key + + +Attributes +'''''''''' + +* ``override-project-caches`` + + Whether this user configuration overrides the project recommendations for + :ref:`artifact caches ` or :ref:`source caches `. + + If this is false (which is the default), then project recommended cache + servers will be observed after user specified caches. + +* ``servers`` + + This is the list of cache servers in the configuration block, every entry + in the block represents a server which will be accessed in the specified order. + + * ``url`` + + Indicates the ``http`` or ``https`` url and optionally the port number of + where the cache server is located. + + * ``instance-name`` + + Instance names separate different shards on the same endpoint (``url``). + + The instance name is optional, and not all cache server implementations support + instance names. The instance name should be given to you by the + service provider of each service. + + * ``type`` + + The type of service you intend to use this cache server for. If unspecified, + the default value for this field is ``all``. + + * ``storage`` + + Use this cache service for storage. + + * ``index`` + + Use this cache service for index content expected to be present in one + or more *storage* services. + + * ``all`` + + Use this cache service for both indexing and storing data. + + * ``push`` + + Set this to ``true`` if you intend to upload data to this cache server. + + Normally this requires additional credentials in the ``auth`` field. + + * ``auth`` + + The :ref:`authentication attributes ` to connect to + this server. + + +.. _config_cache_server_list: + +Cache server lists +'''''''''''''''''' +Cache servers are always specified as *lists* in the configuration, this allows +*index* and *storage* services to be declared separately, and also allows for +some redundancy. + +**Example:** + +.. code:: yaml + + - url: https://cache-server-1.com/index + type: index + - url: https://cache-server-1.com/storage + type: storage + - url: https://cache-server-2.com + type: all + +When downloading data from a cache server, BuildStream will iterate over each +*index* service one by one until it finds the reference to the data it is looking +for, and then it will iterate over each *storage* service one by one, downloading +the referenced data until all data is downloaded. + +When uploading data to a cache server, BuildStream will first upload the data to +each *storage* service which was configured with the ``push`` attribute, and +upon successful upload, it will proceed to upload the references to the uploaded +data to each *index* service in the list. + + +.. _config_artifact_caches: + +Artifact cache servers +~~~~~~~~~~~~~~~~~~~~~~ +Using artifact :ref:`cache servers ` is an essential means of +*build avoidance*, as it will allow you to avoid building an element which has already +been built and uploaded to a common artifact server. + +Artifact cache servers can be declared in different ways, with differing priorities. + + +Command line +'''''''''''' +Various commands which involve connecting to artifact servers allow +:ref:`specifying remotes `, remotes specified +on the command line replace all user configuration. + + +Global caches +''''''''''''' +To declare the global artifact server list, use the ``artifacts`` key at the +toplevel of the user configuration. + +.. code:: yaml + + # + # Configure a global artifact server for pushing and pulling artifacts + # + artifacts: + override-project-caches: false + servers: + - url: https://artifacts.com/artifacts:11001 + push: true + auth: + server-cert: server.crt + client-cert: client.crt + client-key: client.key + + +Project overrides +''''''''''''''''' +To declare artifact servers lists for individual projects, declare them +in the :ref:`project specific section ` of +the user configuration. + +Artifact server lists declared in this section will only be used for +elements belonging to the specified project, and will be used instead of +artifact cache servers declared in the global caches. + +.. code:: yaml + + # + # Configure an artifact server for pushing and pulling artifacts from project "foo" + # + projects: + foo: + artifacts: + override-project-caches: false + servers: + - url: https://artifacts.com/artifacts:11001 + push: true + auth: + server-cert: server.crt + client-cert: client.crt + client-key: client.key + + +Project recommendations +''''''''''''''''''''''' +Projects can :ref:`recommend artifact cache servers ` in their +individual project configuration files. + +These will only be used for elements belonging to their respective projects, and +are the lowest priority configuration. + + +.. _config_source_caches: + +Source cache servers +~~~~~~~~~~~~~~~~~~~~ +Using source :ref:`cache servers ` enables BuildStream to cache +source code referred to by your project and share those sources with peers who have +access to the same source cache server. + +This can optimize your build times in the case that it is determined that an element needs +to be rebuilt because of changes in the dependency graph, as BuildStream will first attempt +to download the source code from the cache server before attempting to obtain it from an +external source, which may suffer higher latencies. + +Source cache servers can be declared in different ways, with differing priorities. + + +Command line +'''''''''''' +Various commands which involve connecting to source cache servers allow +:ref:`specifying remotes `, remotes specified +on the command line replace all user configuration. + + +Global caches +''''''''''''' +To declare the global source cache server list, use the ``source-caches`` key at the +toplevel of the user configuration. + +.. code:: yaml + + # + # Configure a global source cache server for pushing and pulling sources + # + source-caches: + override-project-caches: false + servers: + - url: https://sources.com/sources:11001 + push: true + auth: + server-cert: server.crt + client-cert: client.crt + client-key: client.key + + +Project overrides +''''''''''''''''' +To declare source cache servers lists for individual projects, declare them +in the :ref:`project specific section ` of +the user configuration. + +Source cache server lists declared in this section will only be used for +elements belonging to the specified project, and will be used instead of +source cache servers declared in the global caches. + +.. code:: yaml + + # + # Configure a source cache server for pushing and pulling sources from project "foo" + # + projects: + foo: + source-caches: + override-project-caches: false + servers: + - url: https://sources.com/sources:11001 + push: true + auth: + server-cert: server.crt + client-cert: client.crt + client-key: client.key + + +Project recommendations +''''''''''''''''''''''' +Projects can :ref:`recommend source cache servers ` in their +individual project configuration files. + +These will only be used for elements belonging to their respective projects, and +are the lowest priority configuration. + + +.. _user_config_remote_execution: + +Remote execution +~~~~~~~~~~~~~~~~ +BuildStream supports building remotely using the +`Google Remote Execution API (REAPI). `_. + +You can configure the remote execution services globally in your user configuration +using the ``remote-execution`` key, like so: + +.. code:: yaml + + remote-execution: + execution-service: + url: http://execution.fallback.example.com:50051 + instance-name: main + storage-service: + url: https://storage.fallback.example.com:11002 + instance-name: main + auth: + server-cert: /keys/server.crt + client-cert: /keys/client.crt + client-key: /keys/client.key + action-cache-service: + url: http://cache.flalback.example.com:50052 + instance-name: main + +Attributes +'''''''''' + +* ``execution-service`` + + A :ref:`service configuration ` specifying + how to connect with the main *execution service*, this service is the main controlling + entity in a remote execution build cluster. + +* ``storage-service`` + + A :ref:`service configuration ` specifying + how to connect with the *Content Addressable Storage* service, this is where build + input and output is stored on the remote execution build cluster. + + This service is compatible with the *storage* service offered by + :ref:`cache servers `. + + This is optional if a ``storage-service`` is configured in the + :ref:`cache configuration `, in which case actual file + contents of build outputs will only be downloaded as needed, e.g. on + ``bst artifact checkout``. + +* ``action-cache-service`` + + A :ref:`service configuration ` specifying + how to connect with the *action cache*, this service stores information about + activities which clients request be performed by workers on the remote execution + build cluster, and results of completed operations. + + This service is optional in a remote execution build cluster, if your remote + execution service provides an action cache, then you should configure it here. + + +.. _user_config_remote_execution_service: + +Remote execution service configuration +'''''''''''''''''''''''''''''''''''''' +Each of the distinct services are described by the same configuration block, +which looks like this: + +.. code:: yaml + + url: https://storage.fallback.example.com:11002 + instance-name: main + auth: + server-cert: /keys/server.crt + client-cert: /keys/client.crt + client-key: /keys/client.key + +**Attributes:** + +* ``url`` + + Indicates the ``http`` or ``https`` url and optionally the port number of + where the service is located. + +* ``instance-name`` + + The instance name is optional. Instance names separate different shards on + the same endpoint (``url``). The instance name should be given to you by the + service provider of each service. + + Not all service providers support instance names. + +* ``auth`` + + The :ref:`authentication attributes ` to connect to + this server. + + +.. _user_config_project_overrides: + +Project specific values +----------------------- +The ``projects`` key can be used to specify project specific configurations, +the supported configurations on a project wide basis are listed here. + + +.. _user_config_strict_mode: + +Strict build plan +~~~~~~~~~~~~~~~~~ +The strict build plan option decides whether you want elements +to rebuild when their dependencies have changed. This is enabled +by default, but recommended to turn off in developer scenarios where +you might want to build a large system and test it quickly after +modifying some low level component. + +**Example** + +.. code:: yaml + + projects: + project-name: + strict: False + + +.. note:: + + It is always possible to override this at invocation time using + the ``--strict`` and ``--no-strict`` command line options. + + +.. _config_mirrors: + +Mirrors +~~~~~~~ +Project defined :ref:`mirrors `, can be overridden +with user configuration. This is helpful when you need to mirror all of the source +code used by subprojects and ensure that your project can be built in perpetuity. + +**Example** + +.. code:: yaml + + projects: + project-name: + mirrors: + - name: middle-earth + aliases: + foo: + - http://www.middle-earth.com/foo/1 + - http://www.middle-earth.com/foo/2 + bar: + - http://www.middle-earth.com/bar/1 + - http://www.middle-earth.com/bar/2 + - name: oz + aliases: + foo: + - http://www.oz.com/foo + bar: + - http://www.oz.com/bar + + +.. _config_default_mirror: + +Default mirror +~~~~~~~~~~~~~~ +When using :ref:`mirrors `, one can specify which +mirror should be used first. + +**Example** + +.. code:: yaml + + projects: + project-name: + default-mirror: oz + + +.. note:: + + It is possible to override this at invocation time using the + ``--default-mirror`` command-line option. + + +Project options +~~~~~~~~~~~~~~~ +One can specify values to use for :ref:`project options ` for the projects +you use here, this avoids needing to specify the options on the command line every time. + +**Example** + +.. code:: yaml + + projects: + + # + # Configure the debug flag offered by `project-name` + # + project-name: + options: + debug-build: True + + +Source cache servers +~~~~~~~~~~~~~~~~~~~~ +As already described in the section concerning configuration of +:ref:`source cache servers `, these can be specified on a per project basis. + + +Artifact cache servers +~~~~~~~~~~~~~~~~~~~~~~ +As already described in the section concerning configuration of +:ref:`artifact cache servers `, these can be specified on a per project basis. + + +Remote execution configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Following the same format as the toplevel :ref:`remote execution configuration `, +the global configuration can be overridden on a per project basis in this project override section. + +**Example** + +.. code:: yaml + + projects: + + project-name: + + # + # If `project-name` is built as the toplevel project in this BuildStream session, + # then use this remote execution configuration instead of any globally defined + # remote execution configuration. + # + remote-execution: + execution-service: + url: http://execution.example.com:50051 + instance-name: main + +.. note:: + + Only one remote execution service will be considered for any invocation of BuildStream. + + If you are building a project which has a junction into another subproject for which you have + specified a project specific remote execution service for in your user configuration, then + it will be ignored in the context of building that toplevel project. + + +.. _config_defaults: + +Default configuration +--------------------- +The default BuildStream configuration is specified here for reference: + + .. literalinclude:: ../../src/buildstream/data/userconfig.yaml + :language: yaml diff --git a/_sources/using_configuring_cache_server.rst.txt b/_sources/using_configuring_cache_server.rst.txt new file mode 100644 index 000000000..1ca3887d6 --- /dev/null +++ b/_sources/using_configuring_cache_server.rst.txt @@ -0,0 +1,69 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _cache_servers: + +Configuring Cache Servers +========================= +BuildStream caches the results of builds in a local artifact cache, and will +avoid building an element if there is a suitable build already present in the +local artifact cache. Similarly it will cache sources and avoid pulling them if +present in the local cache. See :ref:`caches ` for more details. + +In addition to the local caches, you can configure one or more remote caches and +BuildStream will then try to pull a suitable object from one of the remotes, +falling back to performing a local build or fetching a source if needed. + +On the client side, cache servers are declared and configured in +:ref:`user configuration `, and since it is typical +for projects to maintain their own cache servers, it is also possible for +projects to provide recommended :ref:`artifact cache servers ` +and :ref:`source cache servers ` through project +configuration, so that downstream users can download from services +provided by upstream projects by default. + + +Setting up a remote cache +------------------------- +BuildStream relies on the `ContentAddressableStorage protocol +`_ +in order to exchange data with remote services, in concert with the `remote asset protocol +`_ +in order to assign symbolic labels (such as :ref:`artifact names `) to identify +stored content. As such, BuildStream is able to function with any implementations of these +two services. + + +Known implementations +--------------------- +Here are some details about known open source implementations of the required protocols + + +Buildbarn +~~~~~~~~~ +The `Buildbarn `_ project provides a remote execution +service implementation for use in build tooling such as BuildStream, `Bazel `_ +and `recc `_, the `bb-storage `_ +and `bb-remote-asset `_ services are tested +to work as cache service for BuildStream's artifact and source caches. + +A simple configuration to spin up the service using `docker compose `_ follows: + +.. literalinclude:: ../../.github/compose/ci.buildbarn.yml + :language: yaml + +Visit the `bb-storage `_ and +`bb-remote-asset `_ project pages to +find more documentation about setting up services with authentication enabled. diff --git a/_sources/using_configuring_remote_execution.rst.txt b/_sources/using_configuring_remote_execution.rst.txt new file mode 100644 index 000000000..b4f05211b --- /dev/null +++ b/_sources/using_configuring_remote_execution.rst.txt @@ -0,0 +1,80 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _remote_execution_servers: + +Remote Execution Servers +======================== +BuildStream supports building remotely using the +`Google Remote Execution API (REAPI). `_, which +has various known implementations. + +Some of these implementations include: + +* `BuildGrid `_ +* `BuildBarn `_ +* `Buildfarm `_ + +These various implementations implement the `Google Remote Execution API (REAPI) +`_ to various degrees as these projects +have different priorities. + +On the client side, the remote execution service to use can be +specified in the :ref:`user configuration `. + + +BuildStream specific requirements +--------------------------------- +In order for BuildStream to work correctly with a remote execution cluster, there +are a couple of requirements that implementation needs to meet. + + +Implementation of platform properties +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The remote execution service must properly implement `platform properties +`_. + +This is crucial because BuildStream needs to be guaranteed the correct operating +system and ISA which it requests from the service. + + +Staging the input root as the filesystem root +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +BuildStream requires that the *input root* given to the remote execution service +be treated as the absolute *filesystem root*. + +This is because BuildStream provides guarantees that all build dependencies, including +the base runtime and compilers, are defined by elements and run within a sandboxed and +isolated build environment, but the `REAPI `_ +was originally developped without this determinism and control in mind. Instead, typically +it is up to the user to configure a cluster to use a docker image to build payloads +with, rather than allowing the REAPI client to control the entire sandbox. + +Unfortunately the ability to dictate that the *input root* be treated as the *filesystem root* +in a container on remote workers in the cluster is not yet standardized in the REAPI protocol. + +.. note:: + + The *input root* is referred to as the ``input_root_digest`` member of the ``Action`` message + as defined in the `protocol `_ + + +Example working configuration +----------------------------- +A simple configuration to spin up the `BuildGrid `_ service using +`docker compose `_ follows: + +.. literalinclude:: ../../.github/compose/ci.buildgrid.yml + :language: yaml diff --git a/_sources/using_developing.rst.txt b/_sources/using_developing.rst.txt new file mode 100644 index 000000000..ad64e4e2b --- /dev/null +++ b/_sources/using_developing.rst.txt @@ -0,0 +1,25 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +Developing +========== +This section covers the features BuildStream provides for users +to develop their integrated products, as well as developping their +modules in the context of an integrated product. + +.. toctree:: + :numbered: + :maxdepth: 1 + + developing/workspaces.rst + developing/strict-mode.rst diff --git a/_sources/using_examples.rst.txt b/_sources/using_examples.rst.txt new file mode 100644 index 000000000..89918b2ca --- /dev/null +++ b/_sources/using_examples.rst.txt @@ -0,0 +1,29 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _examples: + +Examples +======== +This page contains documentation for real examples of BuildStream projects, +described step by step. All run under CI, so you can trust they are +maintained and work as expected. + + +.. toctree:: + :maxdepth: 1 + + examples/tar-mirror + examples/git-mirror diff --git a/_sources/using_handling_files.rst.txt b/_sources/using_handling_files.rst.txt new file mode 100644 index 000000000..3899435f0 --- /dev/null +++ b/_sources/using_handling_files.rst.txt @@ -0,0 +1,25 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +Handling files +============== +This section covers the various aspects related to finer grained +handling of files. + +.. toctree:: + :numbered: + :maxdepth: 1 + + handling-files/composition.rst + handling-files/filtering.rst + handling-files/overlaps.rst diff --git a/_sources/using_junctions.rst.txt b/_sources/using_junctions.rst.txt new file mode 100644 index 000000000..a48747d87 --- /dev/null +++ b/_sources/using_junctions.rst.txt @@ -0,0 +1,27 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +Combining projects +================== +This section covers the features which allow BuildStream projects to +depend on eachother. + +.. toctree:: + :numbered: + :maxdepth: 1 + + junctions/junction-elements + junctions/junction-workspaces + junctions/junction-includes diff --git a/_sources/using_tutorial.rst.txt b/_sources/using_tutorial.rst.txt new file mode 100644 index 000000000..7e5139f3a --- /dev/null +++ b/_sources/using_tutorial.rst.txt @@ -0,0 +1,32 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +.. _tutorial: + +Getting started +=============== +This is a step by step walkthrough meant help the user quickly get +familiar with the fundamentals of creating and using BuildStream +projects. + +.. toctree:: + :numbered: + :maxdepth: 1 + + tutorial/first-project + tutorial/running-commands + tutorial/autotools + tutorial/integration-commands + tutorial/directives diff --git a/_static/_sphinx_javascript_frameworks_compat.js b/_static/_sphinx_javascript_frameworks_compat.js new file mode 100644 index 000000000..81415803e --- /dev/null +++ b/_static/_sphinx_javascript_frameworks_compat.js @@ -0,0 +1,123 @@ +/* Compatability shim for jQuery and underscores.js. + * + * Copyright Sphinx contributors + * Released under the two clause BSD licence + */ + +/** + * small helper function to urldecode strings + * + * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#Decoding_query_parameters_from_a_URL + */ +jQuery.urldecode = function(x) { + if (!x) { + return x + } + return decodeURIComponent(x.replace(/\+/g, ' ')); +}; + +/** + * small helper function to urlencode strings + */ +jQuery.urlencode = encodeURIComponent; + +/** + * This function returns the parsed url parameters of the + * current request. Multiple values per key are supported, + * it will always return arrays of strings for the value parts. + */ +jQuery.getQueryParameters = function(s) { + if (typeof s === 'undefined') + s = document.location.search; + var parts = s.substr(s.indexOf('?') + 1).split('&'); + var result = {}; + for (var i = 0; i < parts.length; i++) { + var tmp = parts[i].split('=', 2); + var key = jQuery.urldecode(tmp[0]); + var value = jQuery.urldecode(tmp[1]); + if (key in result) + result[key].push(value); + else + result[key] = [value]; + } + return result; +}; + +/** + * highlight a given string on a jquery object by wrapping it in + * span elements with the given class name. + */ +jQuery.fn.highlightText = function(text, className) { + function highlight(node, addItems) { + if (node.nodeType === 3) { + var val = node.nodeValue; + var pos = val.toLowerCase().indexOf(text); + if (pos >= 0 && + !jQuery(node.parentNode).hasClass(className) && + !jQuery(node.parentNode).hasClass("nohighlight")) { + var span; + var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg"); + if (isInSVG) { + span = document.createElementNS("http://www.w3.org/2000/svg", "tspan"); + } else { + span = document.createElement("span"); + span.className = className; + } + span.appendChild(document.createTextNode(val.substr(pos, text.length))); + node.parentNode.insertBefore(span, node.parentNode.insertBefore( + document.createTextNode(val.substr(pos + text.length)), + node.nextSibling)); + node.nodeValue = val.substr(0, pos); + if (isInSVG) { + var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect"); + var bbox = node.parentElement.getBBox(); + rect.x.baseVal.value = bbox.x; + rect.y.baseVal.value = bbox.y; + rect.width.baseVal.value = bbox.width; + rect.height.baseVal.value = bbox.height; + rect.setAttribute('class', className); + addItems.push({ + "parent": node.parentNode, + "target": rect}); + } + } + } + else if (!jQuery(node).is("button, select, textarea")) { + jQuery.each(node.childNodes, function() { + highlight(this, addItems); + }); + } + } + var addItems = []; + var result = this.each(function() { + highlight(this, addItems); + }); + for (var i = 0; i < addItems.length; ++i) { + jQuery(addItems[i].parent).before(addItems[i].target); + } + return result; +}; + +/* + * backward compatibility for jQuery.browser + * This will be supported until firefox bug is fixed. + */ +if (!jQuery.browser) { + jQuery.uaMatch = function(ua) { + ua = ua.toLowerCase(); + + var match = /(chrome)[ \/]([\w.]+)/.exec(ua) || + /(webkit)[ \/]([\w.]+)/.exec(ua) || + /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) || + /(msie) ([\w.]+)/.exec(ua) || + ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) || + []; + + return { + browser: match[ 1 ] || "", + version: match[ 2 ] || "0" + }; + }; + jQuery.browser = {}; + jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true; +} diff --git a/_static/arch-datamodel-context.svg b/_static/arch-datamodel-context.svg new file mode 100644 index 000000000..b22fbf33e --- /dev/null +++ b/_static/arch-datamodel-context.svg @@ -0,0 +1,211 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Command Line ArgumentsArguments specified on the command line have the final say for any user preference.User ConfigurationThe user configuration file overrides any of the previous settings.Project RecommendationsWhile we try to keep project data and user preferences as separate as possible, in some cases it is convenient for the project to suggest a value, such as which artifact server to use.Default ConfigurationBuildStream ships an internal YAML file with the same structure as the user configuration file, this file specifies all of the default configurations. + + + + + + + + + + + + + Default Configuration + + + + + + + + + + + + + Project Recommend + + + + + + + + + + + + + User Configuration + + + + + + + + + + + + + Command Line Arguments + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_static/arch-datamodel-element-composition.svg b/_static/arch-datamodel-element-composition.svg new file mode 100644 index 000000000..985fe7213 --- /dev/null +++ b/_static/arch-datamodel-element-composition.svg @@ -0,0 +1,463 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BuildStream Core DataPublic Domain Data + + + + + + + + + + + + + Core Defaults + + + + + + + + + + + + + Element Declaration + + + + + + + + + + + + + Project Configuration + + + + + + + + + + + + + Element Defaults + + + + + + + + + + + + + Project Overrides + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Element Configuration Data + + + + + + + + + + + + + Element Declaration + + + + + + + + + + + + + Element Defaults + + + + + + + + + + + + + Project Overrides + + + + + + + + + + + + + + + + + + + + Element Configuration Data is composited in the same way as Core Data and Public Data, except that it is not understood by the project configuration so there are no defaults. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Composited the first time that a Element of a given “kind” is instantiated.This is cached on the given Element’s class data for later reuse. + + + + + + Composited every time against the cached class data below every time an Element is instantiated. + + + + + + + \ No newline at end of file diff --git a/_static/arch-datamodel-element.svg b/_static/arch-datamodel-element.svg new file mode 100644 index 000000000..ab75414da --- /dev/null +++ b/_static/arch-datamodel-element.svg @@ -0,0 +1,183 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BuildStream Core DataCore data is understood by the BuildStream core. This includes the list of source declarations which are components of the Element.Element Configuration DataElement Configuration Data is defined by Element plugin classes. This data is internal to a given plugin.Public Domain DataPublic Domain Data is free form and not validated, this provides a way to declare data that is intended to be read back and processed by reverse dependencies. + + + + + + Elementkind: autotoolsdescription: Optional description about foodepends:- gtk+.bst- clutter.bstsources:- <source declarations here>config: configure-commands: (<): - cp %{datadir}/automake-*/config.{sub,guess} .public: bst: integration-commands: - update-flying-pony-cache -f ${datadir}/ponies + + + + + + + \ No newline at end of file diff --git a/_static/arch-datamodel-source-composition.svg b/_static/arch-datamodel-source-composition.svg new file mode 100644 index 000000000..60de23620 --- /dev/null +++ b/_static/arch-datamodel-source-composition.svg @@ -0,0 +1,255 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BuildStream Core DataSource Configuration Data + + + + + + + + + + + + + Source Declaration + + + + + + + + + + + + + Project Overrides + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Composited the first time that a Source of a given “kind” is instantiated.This is cached on the given Source’s class data for later reuse.Note: In the case of Sources, defaults are provided in code. + + + + + + Composited every time against the cached class data below every time an Source is instantiated. + + + + + + + + + + + + + Source Defaults + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_static/arch-datamodel-source.svg b/_static/arch-datamodel-source.svg new file mode 100644 index 000000000..55e303311 --- /dev/null +++ b/_static/arch-datamodel-source.svg @@ -0,0 +1,167 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BuildStream Core DataCore data understood by BuildStream.Source Configuration DataSource Configuration Data is defined by Source plugin classes. + + + + + + Sourcekind: gitdirectory: sub/diruri: upstream:foo.gittrack: masterref: bbf775301a08b9a578ef7f647bc35fe13e816241 + + + + + + + \ No newline at end of file diff --git a/_static/arch-dependency-model-build.svg b/_static/arch-dependency-model-build.svg new file mode 100644 index 000000000..67bc5b4d2 --- /dev/null +++ b/_static/arch-dependency-model-build.svg @@ -0,0 +1,267 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Build scope for the Application element + + + + + + + + + + + + + Application + + + + + + + + + + + + + ServiceRequired at runtime + + + + + + + + + + + + + CompilerRequired at build time + + + + + + + + + + + + + Base RuntimeRequired at all times + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Service DefinitionXML files needed to build the Service + + + + + + + + + + + + + + + + + + + + BootstrapSelf Hosting Compiler + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_static/arch-dependency-model-runtime.svg b/_static/arch-dependency-model-runtime.svg new file mode 100644 index 000000000..dc758c844 --- /dev/null +++ b/_static/arch-dependency-model-runtime.svg @@ -0,0 +1,266 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Runtime scope for the Application element + + + + + + + + + + + + + Application + + + + + + + + + + + + + ServiceRequired at runtime + + + + + + + + + + + + + CompilerRequired at build time + + + + + + + + + + + + + Base RuntimeRequired at all times + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Service DefinitionXML files needed to build the Service + + + + + + + + + + + + + + + + + + + + BootstrapSelf Hosting Compiler + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_static/arch-dependency-model.svg b/_static/arch-dependency-model.svg new file mode 100644 index 000000000..3c755b023 --- /dev/null +++ b/_static/arch-dependency-model.svg @@ -0,0 +1,299 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Application + + + + + + + + + + + + + ServiceRequired at runtime + + + + + + + + + + + + + CompilerRequired at build time + + + + + + + + + + + + + Base RuntimeRequired at all times + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + LegendBuild DependencyRuntime DependencyBuild & Runtime Dependency + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Service DefinitionXML files needed to build the Service + + + + + + + + + + + + + + + + + + + + BootstrapSelf Hosting Compiler + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_static/arch-overview.svg b/_static/arch-overview.svg new file mode 100644 index 000000000..738a6a800 --- /dev/null +++ b/_static/arch-overview.svg @@ -0,0 +1,2053 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Frontend + + + + + + + + + + + + + Main Entry Point (cli.py)Implements command line interface + + + + + + + + + + + + + Main Application State (App)Initializes the “Stream”, handles logging and user interactions + + + + + + + + + + + + + + + + + + + + Loggerwidget.py + + + + + + + + + + + + + Status Barstatus.py + + + + + + + + + + + + + Completionscomplete.py + + + + + + + + + + + + + Color profilesprofile.py + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Core, Frontend facing API + + + + + + + + + + + + + ContextUser configuration, CLI arguments + + + + + + + + + + + + + + + + + + + + ProjectLoaded project.conf configuration + + + + + + StreamLoading of the project, implementation of all main user facing commands in the CLI + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Core Internals + + + + + + + + + + + + + PlatformAbstract class providing platform specific routines. + + + + + + The Frontend implements the main command line interface, status bar, master log formatting, and any interactivity with the user. This is cleanly separated from the core BuildStream application. + + + + + + + + + + + + + Scheduler(Scheduling of element processing jobs)The Scheduler is in charge of processing elements in job queues. + + + + + + + + + + + + + Job(Abstract tasks) + + + + + + + + + + + + + Queue(Abstract job queues) + + + + + + + + + + + + + Loader(Load the elements in a project)The loader loads elements and sources from the project directory, transforming them into data structures ready for Element and Source instantiation + + + + + + + + + + + + + LoadElement, Dependency(Loader data model) + + + + + + The Frontend creates the toplevel Project, Context and Stream, and receives Elements and Sources by way of loading the Stream. + + + + + + These represent the main active internal components which sit below the frontend facing business logic in terms of module stacking order. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + OptionsLoads and resolves project options, and processes conditional statements + + + + + + + + + + + + + VariablesResolves a collection of variables, and performs substitutions on strings containing variables + + + + + + + + + + + + + IncludesProcesses include statements and composites the resulting dictionaries + + + + + + YAML Parsing Utilities + + + + + + + + + + + + + YAMLLow level utilities for parsing YAML, storing the provenance of loaded values for error reporting, composition of dictionaries and validation of loaded values + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + utils.pyVarious miscellaneous utilities, includes the directory staging mechanics + + + + + + + + + + + + + signals.pyUtilities and context managers for dealing with UNIX signal handling + + + + + + + + + + + + + utils.pyVarious miscellaneous utilities, includes the directory staging mechanics + + + + + + + + + + + + + WorkspacesHelper object for (de)serialization of the workspace configuration + + + + + + Low Level Subsystems + + + + + + + + + + + + + Element, Source, and PluginData Model + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CAS Server + + + + + + + + + + + + + CasServerContent Addressable Storage Server + + + + + + The CAS Server only needs to interact with Core Internals and lower levels of the stack. + + + + + + + + + + + + + CasCacheContent Addressable Storage implementation, used by the artifact cache and CAS server + + + + + + + + + + + + + FUSE / SafeHardlinksThe Fuse layer implements a copy on write hardlink experience for the Sandbox implementations + + + + + + + + + + + + + SourceFactory / ElementFactoryLow level factory for instantiating Source and Element plugins, one plugin namespace per project + + + + + + + + + + + + + ArtifactCacheArtifact Storage Module + + + + + + + + + + + + + SandboxAbstract Class for running commands in the isolated build environment + + + + + + + + \ No newline at end of file diff --git a/_static/arch-program-flow.svg b/_static/arch-program-flow.svg new file mode 100644 index 000000000..ba196a90d --- /dev/null +++ b/_static/arch-program-flow.svg @@ -0,0 +1,1819 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Parse Command Line + + + + + + + + + + + + + + + + + + + + Load Context + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Enter Application Context + + + + + + + + + + + + + + + + + + + + Load Platform + + + + + + + + + + + + + + + + + + + + ArtifactCache Preflight + + + + + + + + + + + + + + + + + + + + Initialize Logging + + + + + + + + + + + + + + + + + + + + + + + + + + + Exit with error + + + + + + + + + + + + + Load Toplevel Project + + + + + + + + + + + + + + + + + + + + Initialize Stream + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Initialize Application + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Run Stream API + + + + + + + + + + + + + Load Elements + + + + + + + + + + + + + + + + + + + + Resolve Keys + + + + + + + + + + + + + + + + + + + + Resolve Cached State + + + + + + + + + + + + + + + + + + + + Construct Scheduler Queues + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + EventSwitch + + + + + + + + + + + + + + + + + + + + Log Line + + + + + + + + + + + + + + + + + + + + Logging Event + + + + + + Interrupt or Error Event + + + + + + + + App isInteractive + + + + + + + + + + + + + Run SchedulerRun the event loop, and process the elements until they have passed through all of the queues, or until there is a terminate condition. + + + + + + + + + + + + + Interactive + + + + + + + + + + + + + Pause Scheduler + + + + + + + + Prompt User + + + + + + + + + + + + + + + + + + + + Resume Scheduler + + + + + + + + + + + + + Set Terminate Condition + + + + + + + + + + + + + + + + + + + + Non Interactive + + + + + + Continue + + + + + + + + + + + + + Set Terminate Condition + + + + + + + + + + + + + Terminate + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Log Session Summary + + + + + + + + + + + + + Clean Exit + + + + + + + + + + + + + Exception + + + + + + Event + + + + + + + \ No newline at end of file diff --git a/_static/arch-remote-execution.svg b/_static/arch-remote-execution.svg new file mode 100644 index 000000000..e55bebb2d --- /dev/null +++ b/_static/arch-remote-execution.svg @@ -0,0 +1,499 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Remote Execution service + + + + + + + + + + + + + Local artifact cache + + + + + + push + + + + + + + + + + + + + + REAPI + + + + + + + + Remote artifact cache 1 + + + + + + + + + + + + + + + + + + + + pull + + + + + + + + Remote artifact cache 2 + + + + + + push + + + + + + + + + + + + + + + + + + + + pull + + + + + + + + + + + + + + Send (initial environments, commands) + + + + + + + Retrieve (finished environments, logs) + + + + + + + + Cache manager(cascache.py) + + + + + + + + Sandbox.run(commands, flags, environment) + + + + + + + + Remote execution cache + + + + + + + + REAPI-compatible build service + + + + + + BuildStream + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Remote execution service implementation + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_static/arch-scheduler-job.svg b/_static/arch-scheduler-job.svg new file mode 100644 index 000000000..9a50135d8 --- /dev/null +++ b/_static/arch-scheduler-job.svg @@ -0,0 +1,750 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Setup signal handlers + + + + + + + + + + + + + Block signals + + + + + + + + + + + + + fork() + + + + + + + + + + + + + Unblock signals + + + + + + + + + + + + + + + + + + + + + + + + + + + Main BuildStream Process + + + + + + Job Process + + + + + + + + + + + + + Initialize subprocess logging redirection + + + + + + + + + + + + + + + + + + + + Unblock selected signals (ready) + + + + + + + + + + + + + + + + + + + + Run the job payload + + + + + + + + + + + + + + + + + + + + Log messages + + + + + + + + + + + + + + + + + + + Start New Job + + + + + + + + + + + + Receive IPC Messages + + + + + + + + + + + + + Report job result and any other state + + + + + + + + + + + + + Collect results on job instance + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Collect exit status + + + + + + + + + + + + + Exit with symbolic exit status + + + + + + + + + + + + Interruptible Job Harness + + + + + + + + + + + + + Inform the job owner that a job completed, hand over the result + + + + + + + + + + + + + + + + + + + + + + + + + + Job ControlSuspend (SIGTSTP)Resume (SIGCONT)Terminate (SIGTERM)Kill (SIGKILL) + + + + + + + + + + + + + Job Exited + + + + + + + + + + + + + This process is a fork() of the main process without execve().This means it has access to the main process data model at the time of the fork(), any modifications made after this point are copy-on-write, and any side effects of the job need to be manually propagated back to the main process through the IPC. + + + + + + + \ No newline at end of file diff --git a/_static/arch-scheduler-queue-ports.svg b/_static/arch-scheduler-queue-ports.svg new file mode 100644 index 000000000..2466551bf --- /dev/null +++ b/_static/arch-scheduler-queue-ports.svg @@ -0,0 +1,318 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Processed + + + + + + + + + + + + + Failed + + + + + + + + + + + + + Skipped + + + + + + + + + + + + + + + + + + + Ready + + + + + + + + + + + + + Skip + + + + + + Queue implementations report a “QueueStatus” for all of the elements which are in the input queue at all times.Skip elements go directly to the output queue without processing, and the scheduler only ever processes the Ready elements. + + + + + + + + + + + + + Wait + + + + + + + + + + + + + Elements In + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Element Processing + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + After elements are either processed or skipped, they move to the Queue’s output queue where the scheduler can pick them up and and move them along to the next Queue.Elements which have passed through a Queue are also kept in status lists for bookkeeping purposes. + + + + + + Input Queue + + + + + + Output Queue + + + + + + + \ No newline at end of file diff --git a/_static/arch-scheduler-queues.svg b/_static/arch-scheduler-queues.svg new file mode 100644 index 000000000..8732b9fc4 --- /dev/null +++ b/_static/arch-scheduler-queues.svg @@ -0,0 +1,488 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Pull + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Fetch + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Build + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Track + + + + + + + + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + + + + + + + + Element + + + + + + Push + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Elements In + + + + + + + + + + + + + + + + + + + + Elements Out + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_static/arch-scheduler-run.svg b/_static/arch-scheduler-run.svg new file mode 100644 index 000000000..e031aa57e --- /dev/null +++ b/_static/arch-scheduler-run.svg @@ -0,0 +1,437 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Scheduler.run() + + + + + + + + + + + + + Wait + + + + + + Run Event Loop + + + + + + + + + + + + + Job Done + + + + + + + + ReadyElementsRemain + + + + + + + + + + + + + QueueJobs + + + + + + + + + + + + + + + + + + + + Yes + + + + + + + + + + + + + + + + + + + + Return + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + No + + + + + + + \ No newline at end of file diff --git a/_static/basic.css b/_static/basic.css new file mode 100644 index 000000000..7ebbd6d07 --- /dev/null +++ b/_static/basic.css @@ -0,0 +1,914 @@ +/* + * Sphinx stylesheet -- basic theme. + */ + +/* -- main layout ----------------------------------------------------------- */ + +div.clearer { + clear: both; +} + +div.section::after { + display: block; + content: ''; + clear: left; +} + +/* -- relbar ---------------------------------------------------------------- */ + +div.related { + width: 100%; + font-size: 90%; +} + +div.related h3 { + display: none; +} + +div.related ul { + margin: 0; + padding: 0 0 0 10px; + list-style: none; +} + +div.related li { + display: inline; +} + +div.related li.right { + float: right; + margin-right: 5px; +} + +/* -- sidebar --------------------------------------------------------------- */ + +div.sphinxsidebarwrapper { + padding: 10px 5px 0 10px; +} + +div.sphinxsidebar { + float: left; + width: 230px; + margin-left: -100%; + font-size: 90%; + word-wrap: break-word; + overflow-wrap : break-word; +} + +div.sphinxsidebar ul { + list-style: none; +} + +div.sphinxsidebar ul ul, +div.sphinxsidebar ul.want-points { + margin-left: 20px; + list-style: square; +} + +div.sphinxsidebar ul ul { + margin-top: 0; + margin-bottom: 0; +} + +div.sphinxsidebar form { + margin-top: 10px; +} + +div.sphinxsidebar input { + border: 1px solid #98dbcc; + font-family: sans-serif; + font-size: 1em; +} + +div.sphinxsidebar #searchbox form.search { + overflow: hidden; +} + +div.sphinxsidebar #searchbox input[type="text"] { + float: left; + width: 80%; + padding: 0.25em; + box-sizing: border-box; +} + +div.sphinxsidebar #searchbox input[type="submit"] { + float: left; + width: 20%; + border-left: none; + padding: 0.25em; + box-sizing: border-box; +} + + +img { + border: 0; + max-width: 100%; +} + +/* -- search page ----------------------------------------------------------- */ + +ul.search { + margin-top: 10px; +} + +ul.search li { + padding: 5px 0; +} + +ul.search li a { + font-weight: bold; +} + +ul.search li p.context { + color: #888; + margin: 2px 0 0 30px; + text-align: left; +} + +ul.keywordmatches li.goodmatch a { + font-weight: bold; +} + +/* -- index page ------------------------------------------------------------ */ + +table.contentstable { + width: 90%; + margin-left: auto; + margin-right: auto; +} + +table.contentstable p.biglink { + line-height: 150%; +} + +a.biglink { + font-size: 1.3em; +} + +span.linkdescr { + font-style: italic; + padding-top: 5px; + font-size: 90%; +} + +/* -- general index --------------------------------------------------------- */ + +table.indextable { + width: 100%; +} + +table.indextable td { + text-align: left; + vertical-align: top; +} + +table.indextable ul { + margin-top: 0; + margin-bottom: 0; + list-style-type: none; +} + +table.indextable > tbody > tr > td > ul { + padding-left: 0em; +} + +table.indextable tr.pcap { + height: 10px; +} + +table.indextable tr.cap { + margin-top: 10px; + background-color: #f2f2f2; +} + +img.toggler { + margin-right: 3px; + margin-top: 3px; + cursor: pointer; +} + +div.modindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +div.genindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +/* -- domain module index --------------------------------------------------- */ + +table.modindextable td { + padding: 2px; + border-collapse: collapse; +} + +/* -- general body styles --------------------------------------------------- */ + +div.body { + min-width: 360px; + max-width: 800px; +} + +div.body p, div.body dd, div.body li, div.body blockquote { + -moz-hyphens: auto; + -ms-hyphens: auto; + -webkit-hyphens: auto; + hyphens: auto; +} + +a.headerlink { + visibility: hidden; +} + +a:visited { + color: #551A8B; +} + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink, +caption:hover > a.headerlink, +p.caption:hover > a.headerlink, +div.code-block-caption:hover > a.headerlink { + visibility: visible; +} + +div.body p.caption { + text-align: inherit; +} + +div.body td { + text-align: left; +} + +.first { + margin-top: 0 !important; +} + +p.rubric { + margin-top: 30px; + font-weight: bold; +} + +img.align-left, figure.align-left, .figure.align-left, object.align-left { + clear: left; + float: left; + margin-right: 1em; +} + +img.align-right, figure.align-right, .figure.align-right, object.align-right { + clear: right; + float: right; + margin-left: 1em; +} + +img.align-center, figure.align-center, .figure.align-center, object.align-center { + display: block; + margin-left: auto; + margin-right: auto; +} + +img.align-default, figure.align-default, .figure.align-default { + display: block; + margin-left: auto; + margin-right: auto; +} + +.align-left { + text-align: left; +} + +.align-center { + text-align: center; +} + +.align-default { + text-align: center; +} + +.align-right { + text-align: right; +} + +/* -- sidebars -------------------------------------------------------------- */ + +div.sidebar, +aside.sidebar { + margin: 0 0 0.5em 1em; + border: 1px solid #ddb; + padding: 7px; + background-color: #ffe; + width: 40%; + float: right; + clear: right; + overflow-x: auto; +} + +p.sidebar-title { + font-weight: bold; +} + +nav.contents, +aside.topic, +div.admonition, div.topic, blockquote { + clear: left; +} + +/* -- topics ---------------------------------------------------------------- */ + +nav.contents, +aside.topic, +div.topic { + border: 1px solid #ccc; + padding: 7px; + margin: 10px 0 10px 0; +} + +p.topic-title { + font-size: 1.1em; + font-weight: bold; + margin-top: 10px; +} + +/* -- admonitions ----------------------------------------------------------- */ + +div.admonition { + margin-top: 10px; + margin-bottom: 10px; + padding: 7px; +} + +div.admonition dt { + font-weight: bold; +} + +p.admonition-title { + margin: 0px 10px 5px 0px; + font-weight: bold; +} + +div.body p.centered { + text-align: center; + margin-top: 25px; +} + +/* -- content of sidebars/topics/admonitions -------------------------------- */ + +div.sidebar > :last-child, +aside.sidebar > :last-child, +nav.contents > :last-child, +aside.topic > :last-child, +div.topic > :last-child, +div.admonition > :last-child { + margin-bottom: 0; +} + +div.sidebar::after, +aside.sidebar::after, +nav.contents::after, +aside.topic::after, +div.topic::after, +div.admonition::after, +blockquote::after { + display: block; + content: ''; + clear: both; +} + +/* -- tables ---------------------------------------------------------------- */ + +table.docutils { + margin-top: 10px; + margin-bottom: 10px; + border: 0; + border-collapse: collapse; +} + +table.align-center { + margin-left: auto; + margin-right: auto; +} + +table.align-default { + margin-left: auto; + margin-right: auto; +} + +table caption span.caption-number { + font-style: italic; +} + +table caption span.caption-text { +} + +table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +th > :first-child, +td > :first-child { + margin-top: 0px; +} + +th > :last-child, +td > :last-child { + margin-bottom: 0px; +} + +/* -- figures --------------------------------------------------------------- */ + +div.figure, figure { + margin: 0.5em; + padding: 0.5em; +} + +div.figure p.caption, figcaption { + padding: 0.3em; +} + +div.figure p.caption span.caption-number, +figcaption span.caption-number { + font-style: italic; +} + +div.figure p.caption span.caption-text, +figcaption span.caption-text { +} + +/* -- field list styles ----------------------------------------------------- */ + +table.field-list td, table.field-list th { + border: 0 !important; +} + +.field-list ul { + margin: 0; + padding-left: 1em; +} + +.field-list p { + margin: 0; +} + +.field-name { + -moz-hyphens: manual; + -ms-hyphens: manual; + -webkit-hyphens: manual; + hyphens: manual; +} + +/* -- hlist styles ---------------------------------------------------------- */ + +table.hlist { + margin: 1em 0; +} + +table.hlist td { + vertical-align: top; +} + +/* -- object description styles --------------------------------------------- */ + +.sig { + font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; +} + +.sig-name, code.descname { + background-color: transparent; + font-weight: bold; +} + +.sig-name { + font-size: 1.1em; +} + +code.descname { + font-size: 1.2em; +} + +.sig-prename, code.descclassname { + background-color: transparent; +} + +.optional { + font-size: 1.3em; +} + +.sig-paren { + font-size: larger; +} + +.sig-param.n { + font-style: italic; +} + +/* C++ specific styling */ + +.sig-inline.c-texpr, +.sig-inline.cpp-texpr { + font-family: unset; +} + +.sig.c .k, .sig.c .kt, +.sig.cpp .k, .sig.cpp .kt { + color: #0033B3; +} + +.sig.c .m, +.sig.cpp .m { + color: #1750EB; +} + +.sig.c .s, .sig.c .sc, +.sig.cpp .s, .sig.cpp .sc { + color: #067D17; +} + + +/* -- other body styles ----------------------------------------------------- */ + +ol.arabic { + list-style: decimal; +} + +ol.loweralpha { + list-style: lower-alpha; +} + +ol.upperalpha { + list-style: upper-alpha; +} + +ol.lowerroman { + list-style: lower-roman; +} + +ol.upperroman { + list-style: upper-roman; +} + +:not(li) > ol > li:first-child > :first-child, +:not(li) > ul > li:first-child > :first-child { + margin-top: 0px; +} + +:not(li) > ol > li:last-child > :last-child, +:not(li) > ul > li:last-child > :last-child { + margin-bottom: 0px; +} + +ol.simple ol p, +ol.simple ul p, +ul.simple ol p, +ul.simple ul p { + margin-top: 0; +} + +ol.simple > li:not(:first-child) > p, +ul.simple > li:not(:first-child) > p { + margin-top: 0; +} + +ol.simple p, +ul.simple p { + margin-bottom: 0; +} + +aside.footnote > span, +div.citation > span { + float: left; +} +aside.footnote > span:last-of-type, +div.citation > span:last-of-type { + padding-right: 0.5em; +} +aside.footnote > p { + margin-left: 2em; +} +div.citation > p { + margin-left: 4em; +} +aside.footnote > p:last-of-type, +div.citation > p:last-of-type { + margin-bottom: 0em; +} +aside.footnote > p:last-of-type:after, +div.citation > p:last-of-type:after { + content: ""; + clear: both; +} + +dl.field-list { + display: grid; + grid-template-columns: fit-content(30%) auto; +} + +dl.field-list > dt { + font-weight: bold; + word-break: break-word; + padding-left: 0.5em; + padding-right: 5px; +} + +dl.field-list > dd { + padding-left: 0.5em; + margin-top: 0em; + margin-left: 0em; + margin-bottom: 0em; +} + +dl { + margin-bottom: 15px; +} + +dd > :first-child { + margin-top: 0px; +} + +dd ul, dd table { + margin-bottom: 10px; +} + +dd { + margin-top: 3px; + margin-bottom: 10px; + margin-left: 30px; +} + +.sig dd { + margin-top: 0px; + margin-bottom: 0px; +} + +.sig dl { + margin-top: 0px; + margin-bottom: 0px; +} + +dl > dd:last-child, +dl > dd:last-child > :last-child { + margin-bottom: 0; +} + +dt:target, span.highlighted { + background-color: #fbe54e; +} + +rect.highlighted { + fill: #fbe54e; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} + +.versionmodified { + font-style: italic; +} + +.system-message { + background-color: #fda; + padding: 5px; + border: 3px solid red; +} + +.footnote:target { + background-color: #ffa; +} + +.line-block { + display: block; + margin-top: 1em; + margin-bottom: 1em; +} + +.line-block .line-block { + margin-top: 0; + margin-bottom: 0; + margin-left: 1.5em; +} + +.guilabel, .menuselection { + font-family: sans-serif; +} + +.accelerator { + text-decoration: underline; +} + +.classifier { + font-style: oblique; +} + +.classifier:before { + font-style: normal; + margin: 0 0.5em; + content: ":"; + display: inline-block; +} + +abbr, acronym { + border-bottom: dotted 1px; + cursor: help; +} + +.translated { + background-color: rgba(207, 255, 207, 0.2) +} + +.untranslated { + background-color: rgba(255, 207, 207, 0.2) +} + +/* -- code displays --------------------------------------------------------- */ + +pre { + overflow: auto; + overflow-y: hidden; /* fixes display issues on Chrome browsers */ +} + +pre, div[class*="highlight-"] { + clear: both; +} + +span.pre { + -moz-hyphens: none; + -ms-hyphens: none; + -webkit-hyphens: none; + hyphens: none; + white-space: nowrap; +} + +div[class*="highlight-"] { + margin: 1em 0; +} + +td.linenos pre { + border: 0; + background-color: transparent; + color: #aaa; +} + +table.highlighttable { + display: block; +} + +table.highlighttable tbody { + display: block; +} + +table.highlighttable tr { + display: flex; +} + +table.highlighttable td { + margin: 0; + padding: 0; +} + +table.highlighttable td.linenos { + padding-right: 0.5em; +} + +table.highlighttable td.code { + flex: 1; + overflow: hidden; +} + +.highlight .hll { + display: block; +} + +div.highlight pre, +table.highlighttable pre { + margin: 0; +} + +div.code-block-caption + div { + margin-top: 0; +} + +div.code-block-caption { + margin-top: 1em; + padding: 2px 5px; + font-size: small; +} + +div.code-block-caption code { + background-color: transparent; +} + +table.highlighttable td.linenos, +span.linenos, +div.highlight span.gp { /* gp: Generic.Prompt */ + user-select: none; + -webkit-user-select: text; /* Safari fallback only */ + -webkit-user-select: none; /* Chrome/Safari */ + -moz-user-select: none; /* Firefox */ + -ms-user-select: none; /* IE10+ */ +} + +div.code-block-caption span.caption-number { + padding: 0.1em 0.3em; + font-style: italic; +} + +div.code-block-caption span.caption-text { +} + +div.literal-block-wrapper { + margin: 1em 0; +} + +code.xref, a code { + background-color: transparent; + font-weight: bold; +} + +h1 code, h2 code, h3 code, h4 code, h5 code, h6 code { + background-color: transparent; +} + +.viewcode-link { + float: right; +} + +.viewcode-back { + float: right; + font-family: sans-serif; +} + +div.viewcode-block:target { + margin: -1px -10px; + padding: 0 10px; +} + +/* -- math display ---------------------------------------------------------- */ + +img.math { + vertical-align: middle; +} + +div.body div.math p { + text-align: center; +} + +span.eqno { + float: right; +} + +span.eqno a.headerlink { + position: absolute; + z-index: 1; +} + +div.math:hover a.headerlink { + visibility: visible; +} + +/* -- printout stylesheet --------------------------------------------------- */ + +@media print { + div.document, + div.documentwrapper, + div.bodywrapper { + margin: 0 !important; + width: 100%; + } + + div.sphinxsidebar, + div.related, + div.footer, + #top-link { + display: none; + } +} \ No newline at end of file diff --git a/_static/css/badge_only.css b/_static/css/badge_only.css new file mode 100644 index 000000000..88ba55b96 --- /dev/null +++ b/_static/css/badge_only.css @@ -0,0 +1 @@ +.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}@font-face{font-family:FontAwesome;font-style:normal;font-weight:400;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#FontAwesome) format("svg")}.fa:before{font-family:FontAwesome;font-style:normal;font-weight:400;line-height:1}.fa:before,a .fa{text-decoration:inherit}.fa:before,a .fa,li .fa{display:inline-block}li .fa-large:before{width:1.875em}ul.fas{list-style-type:none;margin-left:2em;text-indent:-.8em}ul.fas li .fa{width:.8em}ul.fas li .fa-large:before{vertical-align:baseline}.fa-book:before,.icon-book:before{content:"\f02d"}.fa-caret-down:before,.icon-caret-down:before{content:"\f0d7"}.fa-caret-up:before,.icon-caret-up:before{content:"\f0d8"}.fa-caret-left:before,.icon-caret-left:before{content:"\f0d9"}.fa-caret-right:before,.icon-caret-right:before{content:"\f0da"}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60}.rst-versions .rst-current-version:after{clear:both;content:"";display:block}.rst-versions .rst-current-version .fa{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions .rst-other-versions .rtd-current-item{font-weight:700}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}}#flyout-search-form{padding:6px} \ No newline at end of file diff --git a/_static/css/fonts/Roboto-Slab-Bold.woff b/_static/css/fonts/Roboto-Slab-Bold.woff new file mode 100644 index 000000000..6cb600001 Binary files /dev/null and b/_static/css/fonts/Roboto-Slab-Bold.woff differ diff --git a/_static/css/fonts/Roboto-Slab-Bold.woff2 b/_static/css/fonts/Roboto-Slab-Bold.woff2 new file mode 100644 index 000000000..7059e2314 Binary files /dev/null and b/_static/css/fonts/Roboto-Slab-Bold.woff2 differ diff --git a/_static/css/fonts/Roboto-Slab-Regular.woff b/_static/css/fonts/Roboto-Slab-Regular.woff new file mode 100644 index 000000000..f815f63f9 Binary files /dev/null and b/_static/css/fonts/Roboto-Slab-Regular.woff differ diff --git a/_static/css/fonts/Roboto-Slab-Regular.woff2 b/_static/css/fonts/Roboto-Slab-Regular.woff2 new file mode 100644 index 000000000..f2c76e5bd Binary files /dev/null and b/_static/css/fonts/Roboto-Slab-Regular.woff2 differ diff --git a/_static/css/fonts/fontawesome-webfont.eot b/_static/css/fonts/fontawesome-webfont.eot new file mode 100644 index 000000000..e9f60ca95 Binary files /dev/null and b/_static/css/fonts/fontawesome-webfont.eot differ diff --git a/_static/css/fonts/fontawesome-webfont.svg b/_static/css/fonts/fontawesome-webfont.svg new file mode 100644 index 000000000..855c845e5 --- /dev/null +++ b/_static/css/fonts/fontawesome-webfont.svg @@ -0,0 +1,2671 @@ + + + + +Created by FontForge 20120731 at Mon Oct 24 17:37:40 2016 + By ,,, +Copyright Dave Gandy 2016. All rights reserved. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_static/css/fonts/fontawesome-webfont.ttf b/_static/css/fonts/fontawesome-webfont.ttf new file mode 100644 index 000000000..35acda2fa Binary files /dev/null and b/_static/css/fonts/fontawesome-webfont.ttf differ diff --git a/_static/css/fonts/fontawesome-webfont.woff b/_static/css/fonts/fontawesome-webfont.woff new file mode 100644 index 000000000..400014a4b Binary files /dev/null and b/_static/css/fonts/fontawesome-webfont.woff differ diff --git a/_static/css/fonts/fontawesome-webfont.woff2 b/_static/css/fonts/fontawesome-webfont.woff2 new file mode 100644 index 000000000..4d13fc604 Binary files /dev/null and b/_static/css/fonts/fontawesome-webfont.woff2 differ diff --git a/_static/css/fonts/lato-bold-italic.woff b/_static/css/fonts/lato-bold-italic.woff new file mode 100644 index 000000000..88ad05b9f Binary files /dev/null and b/_static/css/fonts/lato-bold-italic.woff differ diff --git a/_static/css/fonts/lato-bold-italic.woff2 b/_static/css/fonts/lato-bold-italic.woff2 new file mode 100644 index 000000000..c4e3d804b Binary files /dev/null and b/_static/css/fonts/lato-bold-italic.woff2 differ diff --git a/_static/css/fonts/lato-bold.woff b/_static/css/fonts/lato-bold.woff new file mode 100644 index 000000000..c6dff51f0 Binary files /dev/null and b/_static/css/fonts/lato-bold.woff differ diff --git a/_static/css/fonts/lato-bold.woff2 b/_static/css/fonts/lato-bold.woff2 new file mode 100644 index 000000000..bb195043c Binary files /dev/null and b/_static/css/fonts/lato-bold.woff2 differ diff --git a/_static/css/fonts/lato-normal-italic.woff b/_static/css/fonts/lato-normal-italic.woff new file mode 100644 index 000000000..76114bc03 Binary files /dev/null and b/_static/css/fonts/lato-normal-italic.woff differ diff --git a/_static/css/fonts/lato-normal-italic.woff2 b/_static/css/fonts/lato-normal-italic.woff2 new file mode 100644 index 000000000..3404f37e2 Binary files /dev/null and b/_static/css/fonts/lato-normal-italic.woff2 differ diff --git a/_static/css/fonts/lato-normal.woff b/_static/css/fonts/lato-normal.woff new file mode 100644 index 000000000..ae1307ff5 Binary files /dev/null and b/_static/css/fonts/lato-normal.woff differ diff --git a/_static/css/fonts/lato-normal.woff2 b/_static/css/fonts/lato-normal.woff2 new file mode 100644 index 000000000..3bf984332 Binary files /dev/null and b/_static/css/fonts/lato-normal.woff2 differ diff --git a/_static/css/theme.css b/_static/css/theme.css new file mode 100644 index 000000000..0f14f1064 --- /dev/null +++ b/_static/css/theme.css @@ -0,0 +1,4 @@ +html{box-sizing:border-box}*,:after,:before{box-sizing:inherit}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}[hidden],audio:not([controls]){display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:active,a:hover{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:700}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;text-decoration:none}ins,mark{color:#000}mark{background:#ff0;font-style:italic;font-weight:700}.rst-content code,.rst-content tt,code,kbd,pre,samp{font-family:monospace,serif;_font-family:courier new,monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:after,q:before{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sup{top:-.5em}sub{bottom:-.25em}dl,ol,ul{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure,form{margin:0}label{cursor:pointer}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type=button],input[type=reset],input[type=submit]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type=search]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}textarea{resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:.2em 0;background:#ccc;color:#000;padding:.2em 0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir br{display:none}.hidden{display:none!important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0 0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media print{body,html,section{background:none!important}*{box-shadow:none!important;text-shadow:none!important;filter:none!important;-ms-filter:none!important}a,a:visited{text-decoration:underline}.ir a:after,a[href^="#"]:after,a[href^="javascript:"]:after{content:""}blockquote,pre{page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}img{max-width:100%!important}@page{margin:.5cm}.rst-content .toctree-wrapper>p.caption,h2,h3,p{orphans:3;widows:3}.rst-content .toctree-wrapper>p.caption,h2,h3{page-break-after:avoid}}.btn,.fa:before,.icon:before,.rst-content .admonition,.rst-content .admonition-title:before,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .code-block-caption .headerlink:before,.rst-content .danger,.rst-content .eqno .headerlink:before,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content p .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-alert,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a button.toctree-expand:before,.wy-menu-vertical li.on a button.toctree-expand:before,.wy-menu-vertical li button.toctree-expand:before,input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week],select,textarea{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}/*! + * Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome + * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License) + */@font-face{font-family:FontAwesome;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713);src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix&v=4.7.0) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#fontawesomeregular) format("svg");font-weight:400;font-style:normal}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content .eqno .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content p .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-menu-vertical li.current>a button.toctree-expand,.wy-menu-vertical li.on a button.toctree-expand,.wy-menu-vertical li button.toctree-expand{display:inline-block;font:normal normal normal 14px/1 FontAwesome;font-size:inherit;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.33333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.28571em;text-align:center}.fa-ul{padding-left:0;margin-left:2.14286em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.14286em;width:2.14286em;top:.14286em;text-align:center}.fa-li.fa-lg{left:-1.85714em}.fa-border{padding:.2em .25em .15em;border:.08em solid #eee;border-radius:.1em}.fa-pull-left{float:left}.fa-pull-right{float:right}.fa-pull-left.icon,.fa.fa-pull-left,.rst-content .code-block-caption .fa-pull-left.headerlink,.rst-content .eqno .fa-pull-left.headerlink,.rst-content .fa-pull-left.admonition-title,.rst-content code.download span.fa-pull-left:first-child,.rst-content dl dt .fa-pull-left.headerlink,.rst-content h1 .fa-pull-left.headerlink,.rst-content h2 .fa-pull-left.headerlink,.rst-content h3 .fa-pull-left.headerlink,.rst-content h4 .fa-pull-left.headerlink,.rst-content h5 .fa-pull-left.headerlink,.rst-content h6 .fa-pull-left.headerlink,.rst-content p .fa-pull-left.headerlink,.rst-content table>caption .fa-pull-left.headerlink,.rst-content tt.download span.fa-pull-left:first-child,.wy-menu-vertical li.current>a button.fa-pull-left.toctree-expand,.wy-menu-vertical li.on a button.fa-pull-left.toctree-expand,.wy-menu-vertical li button.fa-pull-left.toctree-expand{margin-right:.3em}.fa-pull-right.icon,.fa.fa-pull-right,.rst-content .code-block-caption .fa-pull-right.headerlink,.rst-content .eqno .fa-pull-right.headerlink,.rst-content .fa-pull-right.admonition-title,.rst-content code.download span.fa-pull-right:first-child,.rst-content dl dt .fa-pull-right.headerlink,.rst-content h1 .fa-pull-right.headerlink,.rst-content h2 .fa-pull-right.headerlink,.rst-content h3 .fa-pull-right.headerlink,.rst-content h4 .fa-pull-right.headerlink,.rst-content h5 .fa-pull-right.headerlink,.rst-content h6 .fa-pull-right.headerlink,.rst-content p .fa-pull-right.headerlink,.rst-content table>caption .fa-pull-right.headerlink,.rst-content tt.download span.fa-pull-right:first-child,.wy-menu-vertical li.current>a button.fa-pull-right.toctree-expand,.wy-menu-vertical li.on a button.fa-pull-right.toctree-expand,.wy-menu-vertical li button.fa-pull-right.toctree-expand{margin-left:.3em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left,.pull-left.icon,.rst-content .code-block-caption .pull-left.headerlink,.rst-content .eqno .pull-left.headerlink,.rst-content .pull-left.admonition-title,.rst-content code.download span.pull-left:first-child,.rst-content dl dt .pull-left.headerlink,.rst-content h1 .pull-left.headerlink,.rst-content h2 .pull-left.headerlink,.rst-content h3 .pull-left.headerlink,.rst-content h4 .pull-left.headerlink,.rst-content h5 .pull-left.headerlink,.rst-content h6 .pull-left.headerlink,.rst-content p .pull-left.headerlink,.rst-content table>caption .pull-left.headerlink,.rst-content tt.download span.pull-left:first-child,.wy-menu-vertical li.current>a button.pull-left.toctree-expand,.wy-menu-vertical li.on a button.pull-left.toctree-expand,.wy-menu-vertical li button.pull-left.toctree-expand{margin-right:.3em}.fa.pull-right,.pull-right.icon,.rst-content .code-block-caption .pull-right.headerlink,.rst-content .eqno .pull-right.headerlink,.rst-content .pull-right.admonition-title,.rst-content code.download span.pull-right:first-child,.rst-content dl dt .pull-right.headerlink,.rst-content h1 .pull-right.headerlink,.rst-content h2 .pull-right.headerlink,.rst-content h3 .pull-right.headerlink,.rst-content h4 .pull-right.headerlink,.rst-content h5 .pull-right.headerlink,.rst-content h6 .pull-right.headerlink,.rst-content p .pull-right.headerlink,.rst-content table>caption .pull-right.headerlink,.rst-content tt.download span.pull-right:first-child,.wy-menu-vertical li.current>a button.pull-right.toctree-expand,.wy-menu-vertical li.on a button.pull-right.toctree-expand,.wy-menu-vertical li button.pull-right.toctree-expand{margin-left:.3em}.fa-spin{-webkit-animation:fa-spin 2s linear infinite;animation:fa-spin 2s linear infinite}.fa-pulse{-webkit-animation:fa-spin 1s steps(8) infinite;animation:fa-spin 1s steps(8) infinite}@-webkit-keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}@keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}.fa-rotate-90{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";-webkit-transform:rotate(90deg);-ms-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";-webkit-transform:rotate(180deg);-ms-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";-webkit-transform:rotate(270deg);-ms-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1)";-webkit-transform:scaleX(-1);-ms-transform:scaleX(-1);transform:scaleX(-1)}.fa-flip-vertical{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";-webkit-transform:scaleY(-1);-ms-transform:scaleY(-1);transform:scaleY(-1)}:root .fa-flip-horizontal,:root .fa-flip-vertical,:root .fa-rotate-90,:root .fa-rotate-180,:root .fa-rotate-270{filter:none}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:""}.fa-music:before{content:""}.fa-search:before,.icon-search:before{content:""}.fa-envelope-o:before{content:""}.fa-heart:before{content:""}.fa-star:before{content:""}.fa-star-o:before{content:""}.fa-user:before{content:""}.fa-film:before{content:""}.fa-th-large:before{content:""}.fa-th:before{content:""}.fa-th-list:before{content:""}.fa-check:before{content:""}.fa-close:before,.fa-remove:before,.fa-times:before{content:""}.fa-search-plus:before{content:""}.fa-search-minus:before{content:""}.fa-power-off:before{content:""}.fa-signal:before{content:""}.fa-cog:before,.fa-gear:before{content:""}.fa-trash-o:before{content:""}.fa-home:before,.icon-home:before{content:""}.fa-file-o:before{content:""}.fa-clock-o:before{content:""}.fa-road:before{content:""}.fa-download:before,.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{content:""}.fa-arrow-circle-o-down:before{content:""}.fa-arrow-circle-o-up:before{content:""}.fa-inbox:before{content:""}.fa-play-circle-o:before{content:""}.fa-repeat:before,.fa-rotate-right:before{content:""}.fa-refresh:before{content:""}.fa-list-alt:before{content:""}.fa-lock:before{content:""}.fa-flag:before{content:""}.fa-headphones:before{content:""}.fa-volume-off:before{content:""}.fa-volume-down:before{content:""}.fa-volume-up:before{content:""}.fa-qrcode:before{content:""}.fa-barcode:before{content:""}.fa-tag:before{content:""}.fa-tags:before{content:""}.fa-book:before,.icon-book:before{content:""}.fa-bookmark:before{content:""}.fa-print:before{content:""}.fa-camera:before{content:""}.fa-font:before{content:""}.fa-bold:before{content:""}.fa-italic:before{content:""}.fa-text-height:before{content:""}.fa-text-width:before{content:""}.fa-align-left:before{content:""}.fa-align-center:before{content:""}.fa-align-right:before{content:""}.fa-align-justify:before{content:""}.fa-list:before{content:""}.fa-dedent:before,.fa-outdent:before{content:""}.fa-indent:before{content:""}.fa-video-camera:before{content:""}.fa-image:before,.fa-photo:before,.fa-picture-o:before{content:""}.fa-pencil:before{content:""}.fa-map-marker:before{content:""}.fa-adjust:before{content:""}.fa-tint:before{content:""}.fa-edit:before,.fa-pencil-square-o:before{content:""}.fa-share-square-o:before{content:""}.fa-check-square-o:before{content:""}.fa-arrows:before{content:""}.fa-step-backward:before{content:""}.fa-fast-backward:before{content:""}.fa-backward:before{content:""}.fa-play:before{content:""}.fa-pause:before{content:""}.fa-stop:before{content:""}.fa-forward:before{content:""}.fa-fast-forward:before{content:""}.fa-step-forward:before{content:""}.fa-eject:before{content:""}.fa-chevron-left:before{content:""}.fa-chevron-right:before{content:""}.fa-plus-circle:before{content:""}.fa-minus-circle:before{content:""}.fa-times-circle:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before{content:""}.fa-check-circle:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before{content:""}.fa-question-circle:before{content:""}.fa-info-circle:before{content:""}.fa-crosshairs:before{content:""}.fa-times-circle-o:before{content:""}.fa-check-circle-o:before{content:""}.fa-ban:before{content:""}.fa-arrow-left:before{content:""}.fa-arrow-right:before{content:""}.fa-arrow-up:before{content:""}.fa-arrow-down:before{content:""}.fa-mail-forward:before,.fa-share:before{content:""}.fa-expand:before{content:""}.fa-compress:before{content:""}.fa-plus:before{content:""}.fa-minus:before{content:""}.fa-asterisk:before{content:""}.fa-exclamation-circle:before,.rst-content .admonition-title:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before{content:""}.fa-gift:before{content:""}.fa-leaf:before{content:""}.fa-fire:before,.icon-fire:before{content:""}.fa-eye:before{content:""}.fa-eye-slash:before{content:""}.fa-exclamation-triangle:before,.fa-warning:before{content:""}.fa-plane:before{content:""}.fa-calendar:before{content:""}.fa-random:before{content:""}.fa-comment:before{content:""}.fa-magnet:before{content:""}.fa-chevron-up:before{content:""}.fa-chevron-down:before{content:""}.fa-retweet:before{content:""}.fa-shopping-cart:before{content:""}.fa-folder:before{content:""}.fa-folder-open:before{content:""}.fa-arrows-v:before{content:""}.fa-arrows-h:before{content:""}.fa-bar-chart-o:before,.fa-bar-chart:before{content:""}.fa-twitter-square:before{content:""}.fa-facebook-square:before{content:""}.fa-camera-retro:before{content:""}.fa-key:before{content:""}.fa-cogs:before,.fa-gears:before{content:""}.fa-comments:before{content:""}.fa-thumbs-o-up:before{content:""}.fa-thumbs-o-down:before{content:""}.fa-star-half:before{content:""}.fa-heart-o:before{content:""}.fa-sign-out:before{content:""}.fa-linkedin-square:before{content:""}.fa-thumb-tack:before{content:""}.fa-external-link:before{content:""}.fa-sign-in:before{content:""}.fa-trophy:before{content:""}.fa-github-square:before{content:""}.fa-upload:before{content:""}.fa-lemon-o:before{content:""}.fa-phone:before{content:""}.fa-square-o:before{content:""}.fa-bookmark-o:before{content:""}.fa-phone-square:before{content:""}.fa-twitter:before{content:""}.fa-facebook-f:before,.fa-facebook:before{content:""}.fa-github:before,.icon-github:before{content:""}.fa-unlock:before{content:""}.fa-credit-card:before{content:""}.fa-feed:before,.fa-rss:before{content:""}.fa-hdd-o:before{content:""}.fa-bullhorn:before{content:""}.fa-bell:before{content:""}.fa-certificate:before{content:""}.fa-hand-o-right:before{content:""}.fa-hand-o-left:before{content:""}.fa-hand-o-up:before{content:""}.fa-hand-o-down:before{content:""}.fa-arrow-circle-left:before,.icon-circle-arrow-left:before{content:""}.fa-arrow-circle-right:before,.icon-circle-arrow-right:before{content:""}.fa-arrow-circle-up:before{content:""}.fa-arrow-circle-down:before{content:""}.fa-globe:before{content:""}.fa-wrench:before{content:""}.fa-tasks:before{content:""}.fa-filter:before{content:""}.fa-briefcase:before{content:""}.fa-arrows-alt:before{content:""}.fa-group:before,.fa-users:before{content:""}.fa-chain:before,.fa-link:before,.icon-link:before{content:""}.fa-cloud:before{content:""}.fa-flask:before{content:""}.fa-cut:before,.fa-scissors:before{content:""}.fa-copy:before,.fa-files-o:before{content:""}.fa-paperclip:before{content:""}.fa-floppy-o:before,.fa-save:before{content:""}.fa-square:before{content:""}.fa-bars:before,.fa-navicon:before,.fa-reorder:before{content:""}.fa-list-ul:before{content:""}.fa-list-ol:before{content:""}.fa-strikethrough:before{content:""}.fa-underline:before{content:""}.fa-table:before{content:""}.fa-magic:before{content:""}.fa-truck:before{content:""}.fa-pinterest:before{content:""}.fa-pinterest-square:before{content:""}.fa-google-plus-square:before{content:""}.fa-google-plus:before{content:""}.fa-money:before{content:""}.fa-caret-down:before,.icon-caret-down:before,.wy-dropdown .caret:before{content:""}.fa-caret-up:before{content:""}.fa-caret-left:before{content:""}.fa-caret-right:before{content:""}.fa-columns:before{content:""}.fa-sort:before,.fa-unsorted:before{content:""}.fa-sort-desc:before,.fa-sort-down:before{content:""}.fa-sort-asc:before,.fa-sort-up:before{content:""}.fa-envelope:before{content:""}.fa-linkedin:before{content:""}.fa-rotate-left:before,.fa-undo:before{content:""}.fa-gavel:before,.fa-legal:before{content:""}.fa-dashboard:before,.fa-tachometer:before{content:""}.fa-comment-o:before{content:""}.fa-comments-o:before{content:""}.fa-bolt:before,.fa-flash:before{content:""}.fa-sitemap:before{content:""}.fa-umbrella:before{content:""}.fa-clipboard:before,.fa-paste:before{content:""}.fa-lightbulb-o:before{content:""}.fa-exchange:before{content:""}.fa-cloud-download:before{content:""}.fa-cloud-upload:before{content:""}.fa-user-md:before{content:""}.fa-stethoscope:before{content:""}.fa-suitcase:before{content:""}.fa-bell-o:before{content:""}.fa-coffee:before{content:""}.fa-cutlery:before{content:""}.fa-file-text-o:before{content:""}.fa-building-o:before{content:""}.fa-hospital-o:before{content:""}.fa-ambulance:before{content:""}.fa-medkit:before{content:""}.fa-fighter-jet:before{content:""}.fa-beer:before{content:""}.fa-h-square:before{content:""}.fa-plus-square:before{content:""}.fa-angle-double-left:before{content:""}.fa-angle-double-right:before{content:""}.fa-angle-double-up:before{content:""}.fa-angle-double-down:before{content:""}.fa-angle-left:before{content:""}.fa-angle-right:before{content:""}.fa-angle-up:before{content:""}.fa-angle-down:before{content:""}.fa-desktop:before{content:""}.fa-laptop:before{content:""}.fa-tablet:before{content:""}.fa-mobile-phone:before,.fa-mobile:before{content:""}.fa-circle-o:before{content:""}.fa-quote-left:before{content:""}.fa-quote-right:before{content:""}.fa-spinner:before{content:""}.fa-circle:before{content:""}.fa-mail-reply:before,.fa-reply:before{content:""}.fa-github-alt:before{content:""}.fa-folder-o:before{content:""}.fa-folder-open-o:before{content:""}.fa-smile-o:before{content:""}.fa-frown-o:before{content:""}.fa-meh-o:before{content:""}.fa-gamepad:before{content:""}.fa-keyboard-o:before{content:""}.fa-flag-o:before{content:""}.fa-flag-checkered:before{content:""}.fa-terminal:before{content:""}.fa-code:before{content:""}.fa-mail-reply-all:before,.fa-reply-all:before{content:""}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:""}.fa-location-arrow:before{content:""}.fa-crop:before{content:""}.fa-code-fork:before{content:""}.fa-chain-broken:before,.fa-unlink:before{content:""}.fa-question:before{content:""}.fa-info:before{content:""}.fa-exclamation:before{content:""}.fa-superscript:before{content:""}.fa-subscript:before{content:""}.fa-eraser:before{content:""}.fa-puzzle-piece:before{content:""}.fa-microphone:before{content:""}.fa-microphone-slash:before{content:""}.fa-shield:before{content:""}.fa-calendar-o:before{content:""}.fa-fire-extinguisher:before{content:""}.fa-rocket:before{content:""}.fa-maxcdn:before{content:""}.fa-chevron-circle-left:before{content:""}.fa-chevron-circle-right:before{content:""}.fa-chevron-circle-up:before{content:""}.fa-chevron-circle-down:before{content:""}.fa-html5:before{content:""}.fa-css3:before{content:""}.fa-anchor:before{content:""}.fa-unlock-alt:before{content:""}.fa-bullseye:before{content:""}.fa-ellipsis-h:before{content:""}.fa-ellipsis-v:before{content:""}.fa-rss-square:before{content:""}.fa-play-circle:before{content:""}.fa-ticket:before{content:""}.fa-minus-square:before{content:""}.fa-minus-square-o:before,.wy-menu-vertical li.current>a button.toctree-expand:before,.wy-menu-vertical li.on a button.toctree-expand:before{content:""}.fa-level-up:before{content:""}.fa-level-down:before{content:""}.fa-check-square:before{content:""}.fa-pencil-square:before{content:""}.fa-external-link-square:before{content:""}.fa-share-square:before{content:""}.fa-compass:before{content:""}.fa-caret-square-o-down:before,.fa-toggle-down:before{content:""}.fa-caret-square-o-up:before,.fa-toggle-up:before{content:""}.fa-caret-square-o-right:before,.fa-toggle-right:before{content:""}.fa-eur:before,.fa-euro:before{content:""}.fa-gbp:before{content:""}.fa-dollar:before,.fa-usd:before{content:""}.fa-inr:before,.fa-rupee:before{content:""}.fa-cny:before,.fa-jpy:before,.fa-rmb:before,.fa-yen:before{content:""}.fa-rouble:before,.fa-rub:before,.fa-ruble:before{content:""}.fa-krw:before,.fa-won:before{content:""}.fa-bitcoin:before,.fa-btc:before{content:""}.fa-file:before{content:""}.fa-file-text:before{content:""}.fa-sort-alpha-asc:before{content:""}.fa-sort-alpha-desc:before{content:""}.fa-sort-amount-asc:before{content:""}.fa-sort-amount-desc:before{content:""}.fa-sort-numeric-asc:before{content:""}.fa-sort-numeric-desc:before{content:""}.fa-thumbs-up:before{content:""}.fa-thumbs-down:before{content:""}.fa-youtube-square:before{content:""}.fa-youtube:before{content:""}.fa-xing:before{content:""}.fa-xing-square:before{content:""}.fa-youtube-play:before{content:""}.fa-dropbox:before{content:""}.fa-stack-overflow:before{content:""}.fa-instagram:before{content:""}.fa-flickr:before{content:""}.fa-adn:before{content:""}.fa-bitbucket:before,.icon-bitbucket:before{content:""}.fa-bitbucket-square:before{content:""}.fa-tumblr:before{content:""}.fa-tumblr-square:before{content:""}.fa-long-arrow-down:before{content:""}.fa-long-arrow-up:before{content:""}.fa-long-arrow-left:before{content:""}.fa-long-arrow-right:before{content:""}.fa-apple:before{content:""}.fa-windows:before{content:""}.fa-android:before{content:""}.fa-linux:before{content:""}.fa-dribbble:before{content:""}.fa-skype:before{content:""}.fa-foursquare:before{content:""}.fa-trello:before{content:""}.fa-female:before{content:""}.fa-male:before{content:""}.fa-gittip:before,.fa-gratipay:before{content:""}.fa-sun-o:before{content:""}.fa-moon-o:before{content:""}.fa-archive:before{content:""}.fa-bug:before{content:""}.fa-vk:before{content:""}.fa-weibo:before{content:""}.fa-renren:before{content:""}.fa-pagelines:before{content:""}.fa-stack-exchange:before{content:""}.fa-arrow-circle-o-right:before{content:""}.fa-arrow-circle-o-left:before{content:""}.fa-caret-square-o-left:before,.fa-toggle-left:before{content:""}.fa-dot-circle-o:before{content:""}.fa-wheelchair:before{content:""}.fa-vimeo-square:before{content:""}.fa-try:before,.fa-turkish-lira:before{content:""}.fa-plus-square-o:before,.wy-menu-vertical li button.toctree-expand:before{content:""}.fa-space-shuttle:before{content:""}.fa-slack:before{content:""}.fa-envelope-square:before{content:""}.fa-wordpress:before{content:""}.fa-openid:before{content:""}.fa-bank:before,.fa-institution:before,.fa-university:before{content:""}.fa-graduation-cap:before,.fa-mortar-board:before{content:""}.fa-yahoo:before{content:""}.fa-google:before{content:""}.fa-reddit:before{content:""}.fa-reddit-square:before{content:""}.fa-stumbleupon-circle:before{content:""}.fa-stumbleupon:before{content:""}.fa-delicious:before{content:""}.fa-digg:before{content:""}.fa-pied-piper-pp:before{content:""}.fa-pied-piper-alt:before{content:""}.fa-drupal:before{content:""}.fa-joomla:before{content:""}.fa-language:before{content:""}.fa-fax:before{content:""}.fa-building:before{content:""}.fa-child:before{content:""}.fa-paw:before{content:""}.fa-spoon:before{content:""}.fa-cube:before{content:""}.fa-cubes:before{content:""}.fa-behance:before{content:""}.fa-behance-square:before{content:""}.fa-steam:before{content:""}.fa-steam-square:before{content:""}.fa-recycle:before{content:""}.fa-automobile:before,.fa-car:before{content:""}.fa-cab:before,.fa-taxi:before{content:""}.fa-tree:before{content:""}.fa-spotify:before{content:""}.fa-deviantart:before{content:""}.fa-soundcloud:before{content:""}.fa-database:before{content:""}.fa-file-pdf-o:before{content:""}.fa-file-word-o:before{content:""}.fa-file-excel-o:before{content:""}.fa-file-powerpoint-o:before{content:""}.fa-file-image-o:before,.fa-file-photo-o:before,.fa-file-picture-o:before{content:""}.fa-file-archive-o:before,.fa-file-zip-o:before{content:""}.fa-file-audio-o:before,.fa-file-sound-o:before{content:""}.fa-file-movie-o:before,.fa-file-video-o:before{content:""}.fa-file-code-o:before{content:""}.fa-vine:before{content:""}.fa-codepen:before{content:""}.fa-jsfiddle:before{content:""}.fa-life-bouy:before,.fa-life-buoy:before,.fa-life-ring:before,.fa-life-saver:before,.fa-support:before{content:""}.fa-circle-o-notch:before{content:""}.fa-ra:before,.fa-rebel:before,.fa-resistance:before{content:""}.fa-empire:before,.fa-ge:before{content:""}.fa-git-square:before{content:""}.fa-git:before{content:""}.fa-hacker-news:before,.fa-y-combinator-square:before,.fa-yc-square:before{content:""}.fa-tencent-weibo:before{content:""}.fa-qq:before{content:""}.fa-wechat:before,.fa-weixin:before{content:""}.fa-paper-plane:before,.fa-send:before{content:""}.fa-paper-plane-o:before,.fa-send-o:before{content:""}.fa-history:before{content:""}.fa-circle-thin:before{content:""}.fa-header:before{content:""}.fa-paragraph:before{content:""}.fa-sliders:before{content:""}.fa-share-alt:before{content:""}.fa-share-alt-square:before{content:""}.fa-bomb:before{content:""}.fa-futbol-o:before,.fa-soccer-ball-o:before{content:""}.fa-tty:before{content:""}.fa-binoculars:before{content:""}.fa-plug:before{content:""}.fa-slideshare:before{content:""}.fa-twitch:before{content:""}.fa-yelp:before{content:""}.fa-newspaper-o:before{content:""}.fa-wifi:before{content:""}.fa-calculator:before{content:""}.fa-paypal:before{content:""}.fa-google-wallet:before{content:""}.fa-cc-visa:before{content:""}.fa-cc-mastercard:before{content:""}.fa-cc-discover:before{content:""}.fa-cc-amex:before{content:""}.fa-cc-paypal:before{content:""}.fa-cc-stripe:before{content:""}.fa-bell-slash:before{content:""}.fa-bell-slash-o:before{content:""}.fa-trash:before{content:""}.fa-copyright:before{content:""}.fa-at:before{content:""}.fa-eyedropper:before{content:""}.fa-paint-brush:before{content:""}.fa-birthday-cake:before{content:""}.fa-area-chart:before{content:""}.fa-pie-chart:before{content:""}.fa-line-chart:before{content:""}.fa-lastfm:before{content:""}.fa-lastfm-square:before{content:""}.fa-toggle-off:before{content:""}.fa-toggle-on:before{content:""}.fa-bicycle:before{content:""}.fa-bus:before{content:""}.fa-ioxhost:before{content:""}.fa-angellist:before{content:""}.fa-cc:before{content:""}.fa-ils:before,.fa-shekel:before,.fa-sheqel:before{content:""}.fa-meanpath:before{content:""}.fa-buysellads:before{content:""}.fa-connectdevelop:before{content:""}.fa-dashcube:before{content:""}.fa-forumbee:before{content:""}.fa-leanpub:before{content:""}.fa-sellsy:before{content:""}.fa-shirtsinbulk:before{content:""}.fa-simplybuilt:before{content:""}.fa-skyatlas:before{content:""}.fa-cart-plus:before{content:""}.fa-cart-arrow-down:before{content:""}.fa-diamond:before{content:""}.fa-ship:before{content:""}.fa-user-secret:before{content:""}.fa-motorcycle:before{content:""}.fa-street-view:before{content:""}.fa-heartbeat:before{content:""}.fa-venus:before{content:""}.fa-mars:before{content:""}.fa-mercury:before{content:""}.fa-intersex:before,.fa-transgender:before{content:""}.fa-transgender-alt:before{content:""}.fa-venus-double:before{content:""}.fa-mars-double:before{content:""}.fa-venus-mars:before{content:""}.fa-mars-stroke:before{content:""}.fa-mars-stroke-v:before{content:""}.fa-mars-stroke-h:before{content:""}.fa-neuter:before{content:""}.fa-genderless:before{content:""}.fa-facebook-official:before{content:""}.fa-pinterest-p:before{content:""}.fa-whatsapp:before{content:""}.fa-server:before{content:""}.fa-user-plus:before{content:""}.fa-user-times:before{content:""}.fa-bed:before,.fa-hotel:before{content:""}.fa-viacoin:before{content:""}.fa-train:before{content:""}.fa-subway:before{content:""}.fa-medium:before{content:""}.fa-y-combinator:before,.fa-yc:before{content:""}.fa-optin-monster:before{content:""}.fa-opencart:before{content:""}.fa-expeditedssl:before{content:""}.fa-battery-4:before,.fa-battery-full:before,.fa-battery:before{content:""}.fa-battery-3:before,.fa-battery-three-quarters:before{content:""}.fa-battery-2:before,.fa-battery-half:before{content:""}.fa-battery-1:before,.fa-battery-quarter:before{content:""}.fa-battery-0:before,.fa-battery-empty:before{content:""}.fa-mouse-pointer:before{content:""}.fa-i-cursor:before{content:""}.fa-object-group:before{content:""}.fa-object-ungroup:before{content:""}.fa-sticky-note:before{content:""}.fa-sticky-note-o:before{content:""}.fa-cc-jcb:before{content:""}.fa-cc-diners-club:before{content:""}.fa-clone:before{content:""}.fa-balance-scale:before{content:""}.fa-hourglass-o:before{content:""}.fa-hourglass-1:before,.fa-hourglass-start:before{content:""}.fa-hourglass-2:before,.fa-hourglass-half:before{content:""}.fa-hourglass-3:before,.fa-hourglass-end:before{content:""}.fa-hourglass:before{content:""}.fa-hand-grab-o:before,.fa-hand-rock-o:before{content:""}.fa-hand-paper-o:before,.fa-hand-stop-o:before{content:""}.fa-hand-scissors-o:before{content:""}.fa-hand-lizard-o:before{content:""}.fa-hand-spock-o:before{content:""}.fa-hand-pointer-o:before{content:""}.fa-hand-peace-o:before{content:""}.fa-trademark:before{content:""}.fa-registered:before{content:""}.fa-creative-commons:before{content:""}.fa-gg:before{content:""}.fa-gg-circle:before{content:""}.fa-tripadvisor:before{content:""}.fa-odnoklassniki:before{content:""}.fa-odnoklassniki-square:before{content:""}.fa-get-pocket:before{content:""}.fa-wikipedia-w:before{content:""}.fa-safari:before{content:""}.fa-chrome:before{content:""}.fa-firefox:before{content:""}.fa-opera:before{content:""}.fa-internet-explorer:before{content:""}.fa-television:before,.fa-tv:before{content:""}.fa-contao:before{content:""}.fa-500px:before{content:""}.fa-amazon:before{content:""}.fa-calendar-plus-o:before{content:""}.fa-calendar-minus-o:before{content:""}.fa-calendar-times-o:before{content:""}.fa-calendar-check-o:before{content:""}.fa-industry:before{content:""}.fa-map-pin:before{content:""}.fa-map-signs:before{content:""}.fa-map-o:before{content:""}.fa-map:before{content:""}.fa-commenting:before{content:""}.fa-commenting-o:before{content:""}.fa-houzz:before{content:""}.fa-vimeo:before{content:""}.fa-black-tie:before{content:""}.fa-fonticons:before{content:""}.fa-reddit-alien:before{content:""}.fa-edge:before{content:""}.fa-credit-card-alt:before{content:""}.fa-codiepie:before{content:""}.fa-modx:before{content:""}.fa-fort-awesome:before{content:""}.fa-usb:before{content:""}.fa-product-hunt:before{content:""}.fa-mixcloud:before{content:""}.fa-scribd:before{content:""}.fa-pause-circle:before{content:""}.fa-pause-circle-o:before{content:""}.fa-stop-circle:before{content:""}.fa-stop-circle-o:before{content:""}.fa-shopping-bag:before{content:""}.fa-shopping-basket:before{content:""}.fa-hashtag:before{content:""}.fa-bluetooth:before{content:""}.fa-bluetooth-b:before{content:""}.fa-percent:before{content:""}.fa-gitlab:before,.icon-gitlab:before{content:""}.fa-wpbeginner:before{content:""}.fa-wpforms:before{content:""}.fa-envira:before{content:""}.fa-universal-access:before{content:""}.fa-wheelchair-alt:before{content:""}.fa-question-circle-o:before{content:""}.fa-blind:before{content:""}.fa-audio-description:before{content:""}.fa-volume-control-phone:before{content:""}.fa-braille:before{content:""}.fa-assistive-listening-systems:before{content:""}.fa-american-sign-language-interpreting:before,.fa-asl-interpreting:before{content:""}.fa-deaf:before,.fa-deafness:before,.fa-hard-of-hearing:before{content:""}.fa-glide:before{content:""}.fa-glide-g:before{content:""}.fa-sign-language:before,.fa-signing:before{content:""}.fa-low-vision:before{content:""}.fa-viadeo:before{content:""}.fa-viadeo-square:before{content:""}.fa-snapchat:before{content:""}.fa-snapchat-ghost:before{content:""}.fa-snapchat-square:before{content:""}.fa-pied-piper:before{content:""}.fa-first-order:before{content:""}.fa-yoast:before{content:""}.fa-themeisle:before{content:""}.fa-google-plus-circle:before,.fa-google-plus-official:before{content:""}.fa-fa:before,.fa-font-awesome:before{content:""}.fa-handshake-o:before{content:""}.fa-envelope-open:before{content:""}.fa-envelope-open-o:before{content:""}.fa-linode:before{content:""}.fa-address-book:before{content:""}.fa-address-book-o:before{content:""}.fa-address-card:before,.fa-vcard:before{content:""}.fa-address-card-o:before,.fa-vcard-o:before{content:""}.fa-user-circle:before{content:""}.fa-user-circle-o:before{content:""}.fa-user-o:before{content:""}.fa-id-badge:before{content:""}.fa-drivers-license:before,.fa-id-card:before{content:""}.fa-drivers-license-o:before,.fa-id-card-o:before{content:""}.fa-quora:before{content:""}.fa-free-code-camp:before{content:""}.fa-telegram:before{content:""}.fa-thermometer-4:before,.fa-thermometer-full:before,.fa-thermometer:before{content:""}.fa-thermometer-3:before,.fa-thermometer-three-quarters:before{content:""}.fa-thermometer-2:before,.fa-thermometer-half:before{content:""}.fa-thermometer-1:before,.fa-thermometer-quarter:before{content:""}.fa-thermometer-0:before,.fa-thermometer-empty:before{content:""}.fa-shower:before{content:""}.fa-bath:before,.fa-bathtub:before,.fa-s15:before{content:""}.fa-podcast:before{content:""}.fa-window-maximize:before{content:""}.fa-window-minimize:before{content:""}.fa-window-restore:before{content:""}.fa-times-rectangle:before,.fa-window-close:before{content:""}.fa-times-rectangle-o:before,.fa-window-close-o:before{content:""}.fa-bandcamp:before{content:""}.fa-grav:before{content:""}.fa-etsy:before{content:""}.fa-imdb:before{content:""}.fa-ravelry:before{content:""}.fa-eercast:before{content:""}.fa-microchip:before{content:""}.fa-snowflake-o:before{content:""}.fa-superpowers:before{content:""}.fa-wpexplorer:before{content:""}.fa-meetup:before{content:""}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content .eqno .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content p .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-dropdown .caret,.wy-inline-validate.wy-inline-validate-danger .wy-input-context,.wy-inline-validate.wy-inline-validate-info .wy-input-context,.wy-inline-validate.wy-inline-validate-success .wy-input-context,.wy-inline-validate.wy-inline-validate-warning .wy-input-context,.wy-menu-vertical li.current>a button.toctree-expand,.wy-menu-vertical li.on a button.toctree-expand,.wy-menu-vertical li button.toctree-expand{font-family:inherit}.fa:before,.icon:before,.rst-content .admonition-title:before,.rst-content .code-block-caption .headerlink:before,.rst-content .eqno .headerlink:before,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content p .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a button.toctree-expand:before,.wy-menu-vertical li.on a button.toctree-expand:before,.wy-menu-vertical li button.toctree-expand:before{font-family:FontAwesome;display:inline-block;font-style:normal;font-weight:400;line-height:1;text-decoration:inherit}.rst-content .code-block-caption a .headerlink,.rst-content .eqno a .headerlink,.rst-content a .admonition-title,.rst-content code.download a span:first-child,.rst-content dl dt a .headerlink,.rst-content h1 a .headerlink,.rst-content h2 a .headerlink,.rst-content h3 a .headerlink,.rst-content h4 a .headerlink,.rst-content h5 a .headerlink,.rst-content h6 a .headerlink,.rst-content p.caption a .headerlink,.rst-content p a .headerlink,.rst-content table>caption a .headerlink,.rst-content tt.download a span:first-child,.wy-menu-vertical li.current>a button.toctree-expand,.wy-menu-vertical li.on a button.toctree-expand,.wy-menu-vertical li a button.toctree-expand,a .fa,a .icon,a .rst-content .admonition-title,a .rst-content .code-block-caption .headerlink,a .rst-content .eqno .headerlink,a .rst-content code.download span:first-child,a .rst-content dl dt .headerlink,a .rst-content h1 .headerlink,a .rst-content h2 .headerlink,a .rst-content h3 .headerlink,a .rst-content h4 .headerlink,a .rst-content h5 .headerlink,a .rst-content h6 .headerlink,a .rst-content p.caption .headerlink,a .rst-content p .headerlink,a .rst-content table>caption .headerlink,a .rst-content tt.download span:first-child,a .wy-menu-vertical li button.toctree-expand{display:inline-block;text-decoration:inherit}.btn .fa,.btn .icon,.btn .rst-content .admonition-title,.btn .rst-content .code-block-caption .headerlink,.btn .rst-content .eqno .headerlink,.btn .rst-content code.download span:first-child,.btn .rst-content dl dt .headerlink,.btn .rst-content h1 .headerlink,.btn .rst-content h2 .headerlink,.btn .rst-content h3 .headerlink,.btn .rst-content h4 .headerlink,.btn .rst-content h5 .headerlink,.btn .rst-content h6 .headerlink,.btn .rst-content p .headerlink,.btn .rst-content table>caption .headerlink,.btn .rst-content tt.download span:first-child,.btn .wy-menu-vertical li.current>a button.toctree-expand,.btn .wy-menu-vertical li.on a button.toctree-expand,.btn .wy-menu-vertical li button.toctree-expand,.nav .fa,.nav .icon,.nav .rst-content .admonition-title,.nav .rst-content .code-block-caption .headerlink,.nav .rst-content .eqno .headerlink,.nav .rst-content code.download span:first-child,.nav .rst-content dl dt .headerlink,.nav .rst-content h1 .headerlink,.nav .rst-content h2 .headerlink,.nav .rst-content h3 .headerlink,.nav .rst-content h4 .headerlink,.nav .rst-content h5 .headerlink,.nav .rst-content h6 .headerlink,.nav .rst-content p .headerlink,.nav .rst-content table>caption .headerlink,.nav .rst-content tt.download span:first-child,.nav .wy-menu-vertical li.current>a button.toctree-expand,.nav .wy-menu-vertical li.on a button.toctree-expand,.nav .wy-menu-vertical li button.toctree-expand,.rst-content .btn .admonition-title,.rst-content .code-block-caption .btn .headerlink,.rst-content .code-block-caption .nav .headerlink,.rst-content .eqno .btn .headerlink,.rst-content .eqno .nav .headerlink,.rst-content .nav .admonition-title,.rst-content code.download .btn span:first-child,.rst-content code.download .nav span:first-child,.rst-content dl dt .btn .headerlink,.rst-content dl dt .nav .headerlink,.rst-content h1 .btn .headerlink,.rst-content h1 .nav .headerlink,.rst-content h2 .btn .headerlink,.rst-content h2 .nav .headerlink,.rst-content h3 .btn .headerlink,.rst-content h3 .nav .headerlink,.rst-content h4 .btn .headerlink,.rst-content h4 .nav .headerlink,.rst-content h5 .btn .headerlink,.rst-content h5 .nav .headerlink,.rst-content h6 .btn .headerlink,.rst-content h6 .nav .headerlink,.rst-content p .btn .headerlink,.rst-content p .nav .headerlink,.rst-content table>caption .btn .headerlink,.rst-content table>caption .nav .headerlink,.rst-content tt.download .btn span:first-child,.rst-content tt.download .nav span:first-child,.wy-menu-vertical li .btn button.toctree-expand,.wy-menu-vertical li.current>a .btn button.toctree-expand,.wy-menu-vertical li.current>a .nav button.toctree-expand,.wy-menu-vertical li .nav button.toctree-expand,.wy-menu-vertical li.on a .btn button.toctree-expand,.wy-menu-vertical li.on a .nav button.toctree-expand{display:inline}.btn .fa-large.icon,.btn .fa.fa-large,.btn .rst-content .code-block-caption .fa-large.headerlink,.btn .rst-content .eqno .fa-large.headerlink,.btn .rst-content .fa-large.admonition-title,.btn .rst-content code.download span.fa-large:first-child,.btn .rst-content dl dt .fa-large.headerlink,.btn .rst-content h1 .fa-large.headerlink,.btn .rst-content h2 .fa-large.headerlink,.btn .rst-content h3 .fa-large.headerlink,.btn .rst-content h4 .fa-large.headerlink,.btn .rst-content h5 .fa-large.headerlink,.btn .rst-content h6 .fa-large.headerlink,.btn .rst-content p .fa-large.headerlink,.btn .rst-content table>caption .fa-large.headerlink,.btn .rst-content tt.download span.fa-large:first-child,.btn .wy-menu-vertical li button.fa-large.toctree-expand,.nav .fa-large.icon,.nav .fa.fa-large,.nav .rst-content .code-block-caption .fa-large.headerlink,.nav .rst-content .eqno .fa-large.headerlink,.nav .rst-content .fa-large.admonition-title,.nav .rst-content code.download span.fa-large:first-child,.nav .rst-content dl dt .fa-large.headerlink,.nav .rst-content h1 .fa-large.headerlink,.nav .rst-content h2 .fa-large.headerlink,.nav .rst-content h3 .fa-large.headerlink,.nav .rst-content h4 .fa-large.headerlink,.nav .rst-content h5 .fa-large.headerlink,.nav .rst-content h6 .fa-large.headerlink,.nav .rst-content p .fa-large.headerlink,.nav .rst-content table>caption .fa-large.headerlink,.nav .rst-content tt.download span.fa-large:first-child,.nav .wy-menu-vertical li button.fa-large.toctree-expand,.rst-content .btn .fa-large.admonition-title,.rst-content .code-block-caption .btn .fa-large.headerlink,.rst-content .code-block-caption .nav .fa-large.headerlink,.rst-content .eqno .btn .fa-large.headerlink,.rst-content .eqno .nav .fa-large.headerlink,.rst-content .nav .fa-large.admonition-title,.rst-content code.download .btn span.fa-large:first-child,.rst-content code.download .nav span.fa-large:first-child,.rst-content dl dt .btn .fa-large.headerlink,.rst-content dl dt .nav .fa-large.headerlink,.rst-content h1 .btn .fa-large.headerlink,.rst-content h1 .nav .fa-large.headerlink,.rst-content h2 .btn .fa-large.headerlink,.rst-content h2 .nav .fa-large.headerlink,.rst-content h3 .btn .fa-large.headerlink,.rst-content h3 .nav .fa-large.headerlink,.rst-content h4 .btn .fa-large.headerlink,.rst-content h4 .nav .fa-large.headerlink,.rst-content h5 .btn .fa-large.headerlink,.rst-content h5 .nav .fa-large.headerlink,.rst-content h6 .btn .fa-large.headerlink,.rst-content h6 .nav .fa-large.headerlink,.rst-content p .btn .fa-large.headerlink,.rst-content p .nav .fa-large.headerlink,.rst-content table>caption .btn .fa-large.headerlink,.rst-content table>caption .nav .fa-large.headerlink,.rst-content tt.download .btn span.fa-large:first-child,.rst-content tt.download .nav span.fa-large:first-child,.wy-menu-vertical li .btn button.fa-large.toctree-expand,.wy-menu-vertical li .nav button.fa-large.toctree-expand{line-height:.9em}.btn .fa-spin.icon,.btn .fa.fa-spin,.btn .rst-content .code-block-caption .fa-spin.headerlink,.btn .rst-content .eqno .fa-spin.headerlink,.btn .rst-content .fa-spin.admonition-title,.btn .rst-content code.download span.fa-spin:first-child,.btn .rst-content dl dt .fa-spin.headerlink,.btn .rst-content h1 .fa-spin.headerlink,.btn .rst-content h2 .fa-spin.headerlink,.btn .rst-content h3 .fa-spin.headerlink,.btn .rst-content h4 .fa-spin.headerlink,.btn .rst-content h5 .fa-spin.headerlink,.btn .rst-content h6 .fa-spin.headerlink,.btn .rst-content p .fa-spin.headerlink,.btn .rst-content table>caption .fa-spin.headerlink,.btn .rst-content tt.download span.fa-spin:first-child,.btn .wy-menu-vertical li button.fa-spin.toctree-expand,.nav .fa-spin.icon,.nav .fa.fa-spin,.nav .rst-content .code-block-caption .fa-spin.headerlink,.nav .rst-content .eqno .fa-spin.headerlink,.nav .rst-content .fa-spin.admonition-title,.nav .rst-content code.download span.fa-spin:first-child,.nav .rst-content dl dt .fa-spin.headerlink,.nav .rst-content h1 .fa-spin.headerlink,.nav .rst-content h2 .fa-spin.headerlink,.nav .rst-content h3 .fa-spin.headerlink,.nav .rst-content h4 .fa-spin.headerlink,.nav .rst-content h5 .fa-spin.headerlink,.nav .rst-content h6 .fa-spin.headerlink,.nav .rst-content p .fa-spin.headerlink,.nav .rst-content table>caption .fa-spin.headerlink,.nav .rst-content tt.download span.fa-spin:first-child,.nav .wy-menu-vertical li button.fa-spin.toctree-expand,.rst-content .btn .fa-spin.admonition-title,.rst-content .code-block-caption .btn .fa-spin.headerlink,.rst-content .code-block-caption .nav .fa-spin.headerlink,.rst-content .eqno .btn .fa-spin.headerlink,.rst-content .eqno .nav .fa-spin.headerlink,.rst-content .nav .fa-spin.admonition-title,.rst-content code.download .btn span.fa-spin:first-child,.rst-content code.download .nav span.fa-spin:first-child,.rst-content dl dt .btn .fa-spin.headerlink,.rst-content dl dt .nav .fa-spin.headerlink,.rst-content h1 .btn .fa-spin.headerlink,.rst-content h1 .nav .fa-spin.headerlink,.rst-content h2 .btn .fa-spin.headerlink,.rst-content h2 .nav .fa-spin.headerlink,.rst-content h3 .btn .fa-spin.headerlink,.rst-content h3 .nav .fa-spin.headerlink,.rst-content h4 .btn .fa-spin.headerlink,.rst-content h4 .nav .fa-spin.headerlink,.rst-content h5 .btn .fa-spin.headerlink,.rst-content h5 .nav .fa-spin.headerlink,.rst-content h6 .btn .fa-spin.headerlink,.rst-content h6 .nav .fa-spin.headerlink,.rst-content p .btn .fa-spin.headerlink,.rst-content p .nav .fa-spin.headerlink,.rst-content table>caption .btn .fa-spin.headerlink,.rst-content table>caption .nav .fa-spin.headerlink,.rst-content tt.download .btn span.fa-spin:first-child,.rst-content tt.download .nav span.fa-spin:first-child,.wy-menu-vertical li .btn button.fa-spin.toctree-expand,.wy-menu-vertical li .nav button.fa-spin.toctree-expand{display:inline-block}.btn.fa:before,.btn.icon:before,.rst-content .btn.admonition-title:before,.rst-content .code-block-caption .btn.headerlink:before,.rst-content .eqno .btn.headerlink:before,.rst-content code.download span.btn:first-child:before,.rst-content dl dt .btn.headerlink:before,.rst-content h1 .btn.headerlink:before,.rst-content h2 .btn.headerlink:before,.rst-content h3 .btn.headerlink:before,.rst-content h4 .btn.headerlink:before,.rst-content h5 .btn.headerlink:before,.rst-content h6 .btn.headerlink:before,.rst-content p .btn.headerlink:before,.rst-content table>caption .btn.headerlink:before,.rst-content tt.download span.btn:first-child:before,.wy-menu-vertical li button.btn.toctree-expand:before{opacity:.5;-webkit-transition:opacity .05s ease-in;-moz-transition:opacity .05s ease-in;transition:opacity .05s ease-in}.btn.fa:hover:before,.btn.icon:hover:before,.rst-content .btn.admonition-title:hover:before,.rst-content .code-block-caption .btn.headerlink:hover:before,.rst-content .eqno .btn.headerlink:hover:before,.rst-content code.download span.btn:first-child:hover:before,.rst-content dl dt .btn.headerlink:hover:before,.rst-content h1 .btn.headerlink:hover:before,.rst-content h2 .btn.headerlink:hover:before,.rst-content h3 .btn.headerlink:hover:before,.rst-content h4 .btn.headerlink:hover:before,.rst-content h5 .btn.headerlink:hover:before,.rst-content h6 .btn.headerlink:hover:before,.rst-content p .btn.headerlink:hover:before,.rst-content table>caption .btn.headerlink:hover:before,.rst-content tt.download span.btn:first-child:hover:before,.wy-menu-vertical li button.btn.toctree-expand:hover:before{opacity:1}.btn-mini .fa:before,.btn-mini .icon:before,.btn-mini .rst-content .admonition-title:before,.btn-mini .rst-content .code-block-caption .headerlink:before,.btn-mini .rst-content .eqno .headerlink:before,.btn-mini .rst-content code.download span:first-child:before,.btn-mini .rst-content dl dt .headerlink:before,.btn-mini .rst-content h1 .headerlink:before,.btn-mini .rst-content h2 .headerlink:before,.btn-mini .rst-content h3 .headerlink:before,.btn-mini .rst-content h4 .headerlink:before,.btn-mini .rst-content h5 .headerlink:before,.btn-mini .rst-content h6 .headerlink:before,.btn-mini .rst-content p .headerlink:before,.btn-mini .rst-content table>caption .headerlink:before,.btn-mini .rst-content tt.download span:first-child:before,.btn-mini .wy-menu-vertical li button.toctree-expand:before,.rst-content .btn-mini .admonition-title:before,.rst-content .code-block-caption .btn-mini .headerlink:before,.rst-content .eqno .btn-mini .headerlink:before,.rst-content code.download .btn-mini span:first-child:before,.rst-content dl dt .btn-mini .headerlink:before,.rst-content h1 .btn-mini .headerlink:before,.rst-content h2 .btn-mini .headerlink:before,.rst-content h3 .btn-mini .headerlink:before,.rst-content h4 .btn-mini .headerlink:before,.rst-content h5 .btn-mini .headerlink:before,.rst-content h6 .btn-mini .headerlink:before,.rst-content p .btn-mini .headerlink:before,.rst-content table>caption .btn-mini .headerlink:before,.rst-content tt.download .btn-mini span:first-child:before,.wy-menu-vertical li .btn-mini button.toctree-expand:before{font-size:14px;vertical-align:-15%}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.wy-alert{padding:12px;line-height:24px;margin-bottom:24px;background:#e7f2fa}.rst-content .admonition-title,.wy-alert-title{font-weight:700;display:block;color:#fff;background:#6ab0de;padding:6px 12px;margin:-12px -12px 12px}.rst-content .danger,.rst-content .error,.rst-content .wy-alert-danger.admonition,.rst-content .wy-alert-danger.admonition-todo,.rst-content .wy-alert-danger.attention,.rst-content .wy-alert-danger.caution,.rst-content .wy-alert-danger.hint,.rst-content .wy-alert-danger.important,.rst-content .wy-alert-danger.note,.rst-content .wy-alert-danger.seealso,.rst-content .wy-alert-danger.tip,.rst-content .wy-alert-danger.warning,.wy-alert.wy-alert-danger{background:#fdf3f2}.rst-content .danger .admonition-title,.rst-content .danger .wy-alert-title,.rst-content .error .admonition-title,.rst-content .error .wy-alert-title,.rst-content .wy-alert-danger.admonition-todo .admonition-title,.rst-content .wy-alert-danger.admonition-todo .wy-alert-title,.rst-content .wy-alert-danger.admonition .admonition-title,.rst-content .wy-alert-danger.admonition .wy-alert-title,.rst-content .wy-alert-danger.attention .admonition-title,.rst-content .wy-alert-danger.attention .wy-alert-title,.rst-content .wy-alert-danger.caution .admonition-title,.rst-content .wy-alert-danger.caution .wy-alert-title,.rst-content .wy-alert-danger.hint .admonition-title,.rst-content .wy-alert-danger.hint .wy-alert-title,.rst-content .wy-alert-danger.important .admonition-title,.rst-content .wy-alert-danger.important .wy-alert-title,.rst-content .wy-alert-danger.note .admonition-title,.rst-content .wy-alert-danger.note .wy-alert-title,.rst-content .wy-alert-danger.seealso .admonition-title,.rst-content .wy-alert-danger.seealso .wy-alert-title,.rst-content .wy-alert-danger.tip .admonition-title,.rst-content .wy-alert-danger.tip .wy-alert-title,.rst-content .wy-alert-danger.warning .admonition-title,.rst-content .wy-alert-danger.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-danger .admonition-title,.wy-alert.wy-alert-danger .rst-content .admonition-title,.wy-alert.wy-alert-danger .wy-alert-title{background:#f29f97}.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .warning,.rst-content .wy-alert-warning.admonition,.rst-content .wy-alert-warning.danger,.rst-content .wy-alert-warning.error,.rst-content .wy-alert-warning.hint,.rst-content .wy-alert-warning.important,.rst-content .wy-alert-warning.note,.rst-content .wy-alert-warning.seealso,.rst-content .wy-alert-warning.tip,.wy-alert.wy-alert-warning{background:#ffedcc}.rst-content .admonition-todo .admonition-title,.rst-content .admonition-todo .wy-alert-title,.rst-content .attention .admonition-title,.rst-content .attention .wy-alert-title,.rst-content .caution .admonition-title,.rst-content .caution .wy-alert-title,.rst-content .warning .admonition-title,.rst-content .warning .wy-alert-title,.rst-content .wy-alert-warning.admonition .admonition-title,.rst-content .wy-alert-warning.admonition .wy-alert-title,.rst-content .wy-alert-warning.danger .admonition-title,.rst-content .wy-alert-warning.danger .wy-alert-title,.rst-content .wy-alert-warning.error .admonition-title,.rst-content .wy-alert-warning.error .wy-alert-title,.rst-content .wy-alert-warning.hint .admonition-title,.rst-content .wy-alert-warning.hint .wy-alert-title,.rst-content .wy-alert-warning.important .admonition-title,.rst-content .wy-alert-warning.important .wy-alert-title,.rst-content .wy-alert-warning.note .admonition-title,.rst-content .wy-alert-warning.note .wy-alert-title,.rst-content .wy-alert-warning.seealso .admonition-title,.rst-content .wy-alert-warning.seealso .wy-alert-title,.rst-content .wy-alert-warning.tip .admonition-title,.rst-content .wy-alert-warning.tip .wy-alert-title,.rst-content .wy-alert.wy-alert-warning .admonition-title,.wy-alert.wy-alert-warning .rst-content .admonition-title,.wy-alert.wy-alert-warning .wy-alert-title{background:#f0b37e}.rst-content .note,.rst-content .seealso,.rst-content .wy-alert-info.admonition,.rst-content .wy-alert-info.admonition-todo,.rst-content .wy-alert-info.attention,.rst-content .wy-alert-info.caution,.rst-content .wy-alert-info.danger,.rst-content .wy-alert-info.error,.rst-content .wy-alert-info.hint,.rst-content .wy-alert-info.important,.rst-content .wy-alert-info.tip,.rst-content .wy-alert-info.warning,.wy-alert.wy-alert-info{background:#e7f2fa}.rst-content .note .admonition-title,.rst-content .note .wy-alert-title,.rst-content .seealso .admonition-title,.rst-content .seealso .wy-alert-title,.rst-content .wy-alert-info.admonition-todo .admonition-title,.rst-content .wy-alert-info.admonition-todo .wy-alert-title,.rst-content .wy-alert-info.admonition .admonition-title,.rst-content .wy-alert-info.admonition .wy-alert-title,.rst-content .wy-alert-info.attention .admonition-title,.rst-content .wy-alert-info.attention .wy-alert-title,.rst-content .wy-alert-info.caution .admonition-title,.rst-content .wy-alert-info.caution .wy-alert-title,.rst-content .wy-alert-info.danger .admonition-title,.rst-content .wy-alert-info.danger .wy-alert-title,.rst-content .wy-alert-info.error .admonition-title,.rst-content .wy-alert-info.error .wy-alert-title,.rst-content .wy-alert-info.hint .admonition-title,.rst-content .wy-alert-info.hint .wy-alert-title,.rst-content .wy-alert-info.important .admonition-title,.rst-content .wy-alert-info.important .wy-alert-title,.rst-content .wy-alert-info.tip .admonition-title,.rst-content .wy-alert-info.tip .wy-alert-title,.rst-content .wy-alert-info.warning .admonition-title,.rst-content .wy-alert-info.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-info .admonition-title,.wy-alert.wy-alert-info .rst-content .admonition-title,.wy-alert.wy-alert-info .wy-alert-title{background:#6ab0de}.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .wy-alert-success.admonition,.rst-content .wy-alert-success.admonition-todo,.rst-content .wy-alert-success.attention,.rst-content .wy-alert-success.caution,.rst-content .wy-alert-success.danger,.rst-content .wy-alert-success.error,.rst-content .wy-alert-success.note,.rst-content .wy-alert-success.seealso,.rst-content .wy-alert-success.warning,.wy-alert.wy-alert-success{background:#dbfaf4}.rst-content .hint .admonition-title,.rst-content .hint .wy-alert-title,.rst-content .important .admonition-title,.rst-content .important .wy-alert-title,.rst-content .tip .admonition-title,.rst-content .tip .wy-alert-title,.rst-content .wy-alert-success.admonition-todo .admonition-title,.rst-content .wy-alert-success.admonition-todo .wy-alert-title,.rst-content .wy-alert-success.admonition .admonition-title,.rst-content .wy-alert-success.admonition .wy-alert-title,.rst-content .wy-alert-success.attention .admonition-title,.rst-content .wy-alert-success.attention .wy-alert-title,.rst-content .wy-alert-success.caution .admonition-title,.rst-content .wy-alert-success.caution .wy-alert-title,.rst-content .wy-alert-success.danger .admonition-title,.rst-content .wy-alert-success.danger .wy-alert-title,.rst-content .wy-alert-success.error .admonition-title,.rst-content .wy-alert-success.error .wy-alert-title,.rst-content .wy-alert-success.note .admonition-title,.rst-content .wy-alert-success.note .wy-alert-title,.rst-content .wy-alert-success.seealso .admonition-title,.rst-content .wy-alert-success.seealso .wy-alert-title,.rst-content .wy-alert-success.warning .admonition-title,.rst-content .wy-alert-success.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-success .admonition-title,.wy-alert.wy-alert-success .rst-content .admonition-title,.wy-alert.wy-alert-success .wy-alert-title{background:#1abc9c}.rst-content .wy-alert-neutral.admonition,.rst-content .wy-alert-neutral.admonition-todo,.rst-content .wy-alert-neutral.attention,.rst-content .wy-alert-neutral.caution,.rst-content .wy-alert-neutral.danger,.rst-content .wy-alert-neutral.error,.rst-content .wy-alert-neutral.hint,.rst-content .wy-alert-neutral.important,.rst-content .wy-alert-neutral.note,.rst-content .wy-alert-neutral.seealso,.rst-content .wy-alert-neutral.tip,.rst-content .wy-alert-neutral.warning,.wy-alert.wy-alert-neutral{background:#f3f6f6}.rst-content .wy-alert-neutral.admonition-todo .admonition-title,.rst-content .wy-alert-neutral.admonition-todo .wy-alert-title,.rst-content .wy-alert-neutral.admonition .admonition-title,.rst-content .wy-alert-neutral.admonition .wy-alert-title,.rst-content .wy-alert-neutral.attention .admonition-title,.rst-content .wy-alert-neutral.attention .wy-alert-title,.rst-content .wy-alert-neutral.caution .admonition-title,.rst-content .wy-alert-neutral.caution .wy-alert-title,.rst-content .wy-alert-neutral.danger .admonition-title,.rst-content .wy-alert-neutral.danger .wy-alert-title,.rst-content .wy-alert-neutral.error .admonition-title,.rst-content .wy-alert-neutral.error .wy-alert-title,.rst-content .wy-alert-neutral.hint .admonition-title,.rst-content .wy-alert-neutral.hint .wy-alert-title,.rst-content .wy-alert-neutral.important .admonition-title,.rst-content .wy-alert-neutral.important .wy-alert-title,.rst-content .wy-alert-neutral.note .admonition-title,.rst-content .wy-alert-neutral.note .wy-alert-title,.rst-content .wy-alert-neutral.seealso .admonition-title,.rst-content .wy-alert-neutral.seealso .wy-alert-title,.rst-content .wy-alert-neutral.tip .admonition-title,.rst-content .wy-alert-neutral.tip .wy-alert-title,.rst-content .wy-alert-neutral.warning .admonition-title,.rst-content .wy-alert-neutral.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-neutral .admonition-title,.wy-alert.wy-alert-neutral .rst-content .admonition-title,.wy-alert.wy-alert-neutral .wy-alert-title{color:#404040;background:#e1e4e5}.rst-content .wy-alert-neutral.admonition-todo a,.rst-content .wy-alert-neutral.admonition a,.rst-content .wy-alert-neutral.attention a,.rst-content .wy-alert-neutral.caution a,.rst-content .wy-alert-neutral.danger a,.rst-content .wy-alert-neutral.error a,.rst-content .wy-alert-neutral.hint a,.rst-content .wy-alert-neutral.important a,.rst-content .wy-alert-neutral.note a,.rst-content .wy-alert-neutral.seealso a,.rst-content .wy-alert-neutral.tip a,.rst-content .wy-alert-neutral.warning a,.wy-alert.wy-alert-neutral a{color:#2980b9}.rst-content .admonition-todo p:last-child,.rst-content .admonition p:last-child,.rst-content .attention p:last-child,.rst-content .caution p:last-child,.rst-content .danger p:last-child,.rst-content .error p:last-child,.rst-content .hint p:last-child,.rst-content .important p:last-child,.rst-content .note p:last-child,.rst-content .seealso p:last-child,.rst-content .tip p:last-child,.rst-content .warning p:last-child,.wy-alert p:last-child{margin-bottom:0}.wy-tray-container{position:fixed;bottom:0;left:0;z-index:600}.wy-tray-container li{display:block;width:300px;background:transparent;color:#fff;text-align:center;box-shadow:0 5px 5px 0 rgba(0,0,0,.1);padding:0 24px;min-width:20%;opacity:0;height:0;line-height:56px;overflow:hidden;-webkit-transition:all .3s ease-in;-moz-transition:all .3s ease-in;transition:all .3s ease-in}.wy-tray-container li.wy-tray-item-success{background:#27ae60}.wy-tray-container li.wy-tray-item-info{background:#2980b9}.wy-tray-container li.wy-tray-item-warning{background:#e67e22}.wy-tray-container li.wy-tray-item-danger{background:#e74c3c}.wy-tray-container li.on{opacity:1;height:56px}@media screen and (max-width:768px){.wy-tray-container{bottom:auto;top:0;width:100%}.wy-tray-container li{width:100%}}button{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle;cursor:pointer;line-height:normal;-webkit-appearance:button;*overflow:visible}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}button[disabled]{cursor:default}.btn{display:inline-block;border-radius:2px;line-height:normal;white-space:nowrap;text-align:center;cursor:pointer;font-size:100%;padding:6px 12px 8px;color:#fff;border:1px solid rgba(0,0,0,.1);background-color:#27ae60;text-decoration:none;font-weight:400;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 2px -1px hsla(0,0%,100%,.5),inset 0 -2px 0 0 rgba(0,0,0,.1);outline-none:false;vertical-align:middle;*display:inline;zoom:1;-webkit-user-drag:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transition:all .1s linear;-moz-transition:all .1s linear;transition:all .1s linear}.btn-hover{background:#2e8ece;color:#fff}.btn:hover{background:#2cc36b;color:#fff}.btn:focus{background:#2cc36b;outline:0}.btn:active{box-shadow:inset 0 -1px 0 0 rgba(0,0,0,.05),inset 0 2px 0 0 rgba(0,0,0,.1);padding:8px 12px 6px}.btn:visited{color:#fff}.btn-disabled,.btn-disabled:active,.btn-disabled:focus,.btn-disabled:hover,.btn:disabled{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:.4;cursor:not-allowed;box-shadow:none}.btn::-moz-focus-inner{padding:0;border:0}.btn-small{font-size:80%}.btn-info{background-color:#2980b9!important}.btn-info:hover{background-color:#2e8ece!important}.btn-neutral{background-color:#f3f6f6!important;color:#404040!important}.btn-neutral:hover{background-color:#e5ebeb!important;color:#404040}.btn-neutral:visited{color:#404040!important}.btn-success{background-color:#27ae60!important}.btn-success:hover{background-color:#295!important}.btn-danger{background-color:#e74c3c!important}.btn-danger:hover{background-color:#ea6153!important}.btn-warning{background-color:#e67e22!important}.btn-warning:hover{background-color:#e98b39!important}.btn-invert{background-color:#222}.btn-invert:hover{background-color:#2f2f2f!important}.btn-link{background-color:transparent!important;color:#2980b9;box-shadow:none;border-color:transparent!important}.btn-link:active,.btn-link:hover{background-color:transparent!important;color:#409ad5!important;box-shadow:none}.btn-link:visited{color:#9b59b6}.wy-btn-group .btn,.wy-control .btn{vertical-align:middle}.wy-btn-group{margin-bottom:24px;*zoom:1}.wy-btn-group:after,.wy-btn-group:before{display:table;content:""}.wy-btn-group:after{clear:both}.wy-dropdown{position:relative;display:inline-block}.wy-dropdown-active .wy-dropdown-menu{display:block}.wy-dropdown-menu{position:absolute;left:0;display:none;float:left;top:100%;min-width:100%;background:#fcfcfc;z-index:100;border:1px solid #cfd7dd;box-shadow:0 2px 2px 0 rgba(0,0,0,.1);padding:12px}.wy-dropdown-menu>dd>a{display:block;clear:both;color:#404040;white-space:nowrap;font-size:90%;padding:0 12px;cursor:pointer}.wy-dropdown-menu>dd>a:hover{background:#2980b9;color:#fff}.wy-dropdown-menu>dd.divider{border-top:1px solid #cfd7dd;margin:6px 0}.wy-dropdown-menu>dd.search{padding-bottom:12px}.wy-dropdown-menu>dd.search input[type=search]{width:100%}.wy-dropdown-menu>dd.call-to-action{background:#e3e3e3;text-transform:uppercase;font-weight:500;font-size:80%}.wy-dropdown-menu>dd.call-to-action:hover{background:#e3e3e3}.wy-dropdown-menu>dd.call-to-action .btn{color:#fff}.wy-dropdown.wy-dropdown-up .wy-dropdown-menu{bottom:100%;top:auto;left:auto;right:0}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu{background:#fcfcfc;margin-top:2px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a{padding:6px 12px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a:hover{background:#2980b9;color:#fff}.wy-dropdown.wy-dropdown-left .wy-dropdown-menu{right:0;left:auto;text-align:right}.wy-dropdown-arrow:before{content:" ";border-bottom:5px solid #f5f5f5;border-left:5px solid transparent;border-right:5px solid transparent;position:absolute;display:block;top:-4px;left:50%;margin-left:-3px}.wy-dropdown-arrow.wy-dropdown-arrow-left:before{left:11px}.wy-form-stacked select{display:block}.wy-form-aligned .wy-help-inline,.wy-form-aligned input,.wy-form-aligned label,.wy-form-aligned select,.wy-form-aligned textarea{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-form-aligned .wy-control-group>label{display:inline-block;vertical-align:middle;width:10em;margin:6px 12px 0 0;float:left}.wy-form-aligned .wy-control{float:left}.wy-form-aligned .wy-control label{display:block}.wy-form-aligned .wy-control select{margin-top:6px}fieldset{margin:0}fieldset,legend{border:0;padding:0}legend{width:100%;white-space:normal;margin-bottom:24px;font-size:150%;*margin-left:-7px}label,legend{display:block}label{margin:0 0 .3125em;color:#333;font-size:90%}input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}.wy-control-group{margin-bottom:24px;max-width:1200px;margin-left:auto;margin-right:auto;*zoom:1}.wy-control-group:after,.wy-control-group:before{display:table;content:""}.wy-control-group:after{clear:both}.wy-control-group.wy-control-group-required>label:after{content:" *";color:#e74c3c}.wy-control-group .wy-form-full,.wy-control-group .wy-form-halves,.wy-control-group .wy-form-thirds{padding-bottom:12px}.wy-control-group .wy-form-full input[type=color],.wy-control-group .wy-form-full input[type=date],.wy-control-group .wy-form-full input[type=datetime-local],.wy-control-group .wy-form-full input[type=datetime],.wy-control-group .wy-form-full input[type=email],.wy-control-group .wy-form-full input[type=month],.wy-control-group .wy-form-full input[type=number],.wy-control-group .wy-form-full input[type=password],.wy-control-group .wy-form-full input[type=search],.wy-control-group .wy-form-full input[type=tel],.wy-control-group .wy-form-full input[type=text],.wy-control-group .wy-form-full input[type=time],.wy-control-group .wy-form-full input[type=url],.wy-control-group .wy-form-full input[type=week],.wy-control-group .wy-form-full select,.wy-control-group .wy-form-halves input[type=color],.wy-control-group .wy-form-halves input[type=date],.wy-control-group .wy-form-halves input[type=datetime-local],.wy-control-group .wy-form-halves input[type=datetime],.wy-control-group .wy-form-halves input[type=email],.wy-control-group .wy-form-halves input[type=month],.wy-control-group .wy-form-halves input[type=number],.wy-control-group .wy-form-halves input[type=password],.wy-control-group .wy-form-halves input[type=search],.wy-control-group .wy-form-halves input[type=tel],.wy-control-group .wy-form-halves input[type=text],.wy-control-group .wy-form-halves input[type=time],.wy-control-group .wy-form-halves input[type=url],.wy-control-group .wy-form-halves input[type=week],.wy-control-group .wy-form-halves select,.wy-control-group .wy-form-thirds input[type=color],.wy-control-group .wy-form-thirds input[type=date],.wy-control-group .wy-form-thirds input[type=datetime-local],.wy-control-group .wy-form-thirds input[type=datetime],.wy-control-group .wy-form-thirds input[type=email],.wy-control-group .wy-form-thirds input[type=month],.wy-control-group .wy-form-thirds input[type=number],.wy-control-group .wy-form-thirds input[type=password],.wy-control-group .wy-form-thirds input[type=search],.wy-control-group .wy-form-thirds input[type=tel],.wy-control-group .wy-form-thirds input[type=text],.wy-control-group .wy-form-thirds input[type=time],.wy-control-group .wy-form-thirds input[type=url],.wy-control-group .wy-form-thirds input[type=week],.wy-control-group .wy-form-thirds select{width:100%}.wy-control-group .wy-form-full{float:left;display:block;width:100%;margin-right:0}.wy-control-group .wy-form-full:last-child{margin-right:0}.wy-control-group .wy-form-halves{float:left;display:block;margin-right:2.35765%;width:48.82117%}.wy-control-group .wy-form-halves:last-child,.wy-control-group .wy-form-halves:nth-of-type(2n){margin-right:0}.wy-control-group .wy-form-halves:nth-of-type(odd){clear:left}.wy-control-group .wy-form-thirds{float:left;display:block;margin-right:2.35765%;width:31.76157%}.wy-control-group .wy-form-thirds:last-child,.wy-control-group .wy-form-thirds:nth-of-type(3n){margin-right:0}.wy-control-group .wy-form-thirds:nth-of-type(3n+1){clear:left}.wy-control-group.wy-control-group-no-input .wy-control,.wy-control-no-input{margin:6px 0 0;font-size:90%}.wy-control-no-input{display:inline-block}.wy-control-group.fluid-input input[type=color],.wy-control-group.fluid-input input[type=date],.wy-control-group.fluid-input input[type=datetime-local],.wy-control-group.fluid-input input[type=datetime],.wy-control-group.fluid-input input[type=email],.wy-control-group.fluid-input input[type=month],.wy-control-group.fluid-input input[type=number],.wy-control-group.fluid-input input[type=password],.wy-control-group.fluid-input input[type=search],.wy-control-group.fluid-input input[type=tel],.wy-control-group.fluid-input input[type=text],.wy-control-group.fluid-input input[type=time],.wy-control-group.fluid-input input[type=url],.wy-control-group.fluid-input input[type=week]{width:100%}.wy-form-message-inline{padding-left:.3em;color:#666;font-size:90%}.wy-form-message{display:block;color:#999;font-size:70%;margin-top:.3125em;font-style:italic}.wy-form-message p{font-size:inherit;font-style:italic;margin-bottom:6px}.wy-form-message p:last-child{margin-bottom:0}input{line-height:normal}input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;*overflow:visible}input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week]{-webkit-appearance:none;padding:6px;display:inline-block;border:1px solid #ccc;font-size:80%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 3px #ddd;border-radius:0;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}input[type=datetime-local]{padding:.34375em .625em}input[disabled]{cursor:default}input[type=checkbox],input[type=radio]{padding:0;margin-right:.3125em;*height:13px;*width:13px}input[type=checkbox],input[type=radio],input[type=search]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}input[type=search]::-webkit-search-cancel-button,input[type=search]::-webkit-search-decoration{-webkit-appearance:none}input[type=color]:focus,input[type=date]:focus,input[type=datetime-local]:focus,input[type=datetime]:focus,input[type=email]:focus,input[type=month]:focus,input[type=number]:focus,input[type=password]:focus,input[type=search]:focus,input[type=tel]:focus,input[type=text]:focus,input[type=time]:focus,input[type=url]:focus,input[type=week]:focus{outline:0;outline:thin dotted\9;border-color:#333}input.no-focus:focus{border-color:#ccc!important}input[type=checkbox]:focus,input[type=file]:focus,input[type=radio]:focus{outline:thin dotted #333;outline:1px auto #129fea}input[type=color][disabled],input[type=date][disabled],input[type=datetime-local][disabled],input[type=datetime][disabled],input[type=email][disabled],input[type=month][disabled],input[type=number][disabled],input[type=password][disabled],input[type=search][disabled],input[type=tel][disabled],input[type=text][disabled],input[type=time][disabled],input[type=url][disabled],input[type=week][disabled]{cursor:not-allowed;background-color:#fafafa}input:focus:invalid,select:focus:invalid,textarea:focus:invalid{color:#e74c3c;border:1px solid #e74c3c}input:focus:invalid:focus,select:focus:invalid:focus,textarea:focus:invalid:focus{border-color:#e74c3c}input[type=checkbox]:focus:invalid:focus,input[type=file]:focus:invalid:focus,input[type=radio]:focus:invalid:focus{outline-color:#e74c3c}input.wy-input-large{padding:12px;font-size:100%}textarea{overflow:auto;vertical-align:top;width:100%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif}select,textarea{padding:.5em .625em;display:inline-block;border:1px solid #ccc;font-size:80%;box-shadow:inset 0 1px 3px #ddd;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}select{border:1px solid #ccc;background-color:#fff}select[multiple]{height:auto}select:focus,textarea:focus{outline:0}input[readonly],select[disabled],select[readonly],textarea[disabled],textarea[readonly]{cursor:not-allowed;background-color:#fafafa}input[type=checkbox][disabled],input[type=radio][disabled]{cursor:not-allowed}.wy-checkbox,.wy-radio{margin:6px 0;color:#404040;display:block}.wy-checkbox input,.wy-radio input{vertical-align:baseline}.wy-form-message-inline{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-input-prefix,.wy-input-suffix{white-space:nowrap;padding:6px}.wy-input-prefix .wy-input-context,.wy-input-suffix .wy-input-context{line-height:27px;padding:0 8px;display:inline-block;font-size:80%;background-color:#f3f6f6;border:1px solid #ccc;color:#999}.wy-input-suffix .wy-input-context{border-left:0}.wy-input-prefix .wy-input-context{border-right:0}.wy-switch{position:relative;display:block;height:24px;margin-top:12px;cursor:pointer}.wy-switch:before{left:0;top:0;width:36px;height:12px;background:#ccc}.wy-switch:after,.wy-switch:before{position:absolute;content:"";display:block;border-radius:4px;-webkit-transition:all .2s ease-in-out;-moz-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.wy-switch:after{width:18px;height:18px;background:#999;left:-3px;top:-3px}.wy-switch span{position:absolute;left:48px;display:block;font-size:12px;color:#ccc;line-height:1}.wy-switch.active:before{background:#1e8449}.wy-switch.active:after{left:24px;background:#27ae60}.wy-switch.disabled{cursor:not-allowed;opacity:.8}.wy-control-group.wy-control-group-error .wy-form-message,.wy-control-group.wy-control-group-error>label{color:#e74c3c}.wy-control-group.wy-control-group-error input[type=color],.wy-control-group.wy-control-group-error input[type=date],.wy-control-group.wy-control-group-error input[type=datetime-local],.wy-control-group.wy-control-group-error input[type=datetime],.wy-control-group.wy-control-group-error input[type=email],.wy-control-group.wy-control-group-error input[type=month],.wy-control-group.wy-control-group-error input[type=number],.wy-control-group.wy-control-group-error input[type=password],.wy-control-group.wy-control-group-error input[type=search],.wy-control-group.wy-control-group-error input[type=tel],.wy-control-group.wy-control-group-error input[type=text],.wy-control-group.wy-control-group-error input[type=time],.wy-control-group.wy-control-group-error input[type=url],.wy-control-group.wy-control-group-error input[type=week],.wy-control-group.wy-control-group-error textarea{border:1px solid #e74c3c}.wy-inline-validate{white-space:nowrap}.wy-inline-validate .wy-input-context{padding:.5em .625em;display:inline-block;font-size:80%}.wy-inline-validate.wy-inline-validate-success .wy-input-context{color:#27ae60}.wy-inline-validate.wy-inline-validate-danger .wy-input-context{color:#e74c3c}.wy-inline-validate.wy-inline-validate-warning .wy-input-context{color:#e67e22}.wy-inline-validate.wy-inline-validate-info .wy-input-context{color:#2980b9}.rotate-90{-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.rotate-180{-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.rotate-270{-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-transform:rotate(270deg);transform:rotate(270deg)}.mirror{-webkit-transform:scaleX(-1);-moz-transform:scaleX(-1);-ms-transform:scaleX(-1);-o-transform:scaleX(-1);transform:scaleX(-1)}.mirror.rotate-90{-webkit-transform:scaleX(-1) rotate(90deg);-moz-transform:scaleX(-1) rotate(90deg);-ms-transform:scaleX(-1) rotate(90deg);-o-transform:scaleX(-1) rotate(90deg);transform:scaleX(-1) rotate(90deg)}.mirror.rotate-180{-webkit-transform:scaleX(-1) rotate(180deg);-moz-transform:scaleX(-1) rotate(180deg);-ms-transform:scaleX(-1) rotate(180deg);-o-transform:scaleX(-1) rotate(180deg);transform:scaleX(-1) rotate(180deg)}.mirror.rotate-270{-webkit-transform:scaleX(-1) rotate(270deg);-moz-transform:scaleX(-1) rotate(270deg);-ms-transform:scaleX(-1) rotate(270deg);-o-transform:scaleX(-1) rotate(270deg);transform:scaleX(-1) rotate(270deg)}@media only screen and (max-width:480px){.wy-form button[type=submit]{margin:.7em 0 0}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=text],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week],.wy-form label{margin-bottom:.3em;display:block}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week]{margin-bottom:0}.wy-form-aligned .wy-control-group label{margin-bottom:.3em;text-align:left;display:block;width:100%}.wy-form-aligned .wy-control{margin:1.5em 0 0}.wy-form-message,.wy-form-message-inline,.wy-form .wy-help-inline{display:block;font-size:80%;padding:6px 0}}@media screen and (max-width:768px){.tablet-hide{display:none}}@media screen and (max-width:480px){.mobile-hide{display:none}}.float-left{float:left}.float-right{float:right}.full-width{width:100%}.rst-content table.docutils,.rst-content table.field-list,.wy-table{border-collapse:collapse;border-spacing:0;empty-cells:show;margin-bottom:24px}.rst-content table.docutils caption,.rst-content table.field-list caption,.wy-table caption{color:#000;font:italic 85%/1 arial,sans-serif;padding:1em 0;text-align:center}.rst-content table.docutils td,.rst-content table.docutils th,.rst-content table.field-list td,.rst-content table.field-list th,.wy-table td,.wy-table th{font-size:90%;margin:0;overflow:visible;padding:8px 16px}.rst-content table.docutils td:first-child,.rst-content table.docutils th:first-child,.rst-content table.field-list td:first-child,.rst-content table.field-list th:first-child,.wy-table td:first-child,.wy-table th:first-child{border-left-width:0}.rst-content table.docutils thead,.rst-content table.field-list thead,.wy-table thead{color:#000;text-align:left;vertical-align:bottom;white-space:nowrap}.rst-content table.docutils thead th,.rst-content table.field-list thead th,.wy-table thead th{font-weight:700;border-bottom:2px solid #e1e4e5}.rst-content table.docutils td,.rst-content table.field-list td,.wy-table td{background-color:transparent;vertical-align:middle}.rst-content table.docutils td p,.rst-content table.field-list td p,.wy-table td p{line-height:18px}.rst-content table.docutils td p:last-child,.rst-content table.field-list td p:last-child,.wy-table td p:last-child{margin-bottom:0}.rst-content table.docutils .wy-table-cell-min,.rst-content table.field-list .wy-table-cell-min,.wy-table .wy-table-cell-min{width:1%;padding-right:0}.rst-content table.docutils .wy-table-cell-min input[type=checkbox],.rst-content table.field-list .wy-table-cell-min input[type=checkbox],.wy-table .wy-table-cell-min input[type=checkbox]{margin:0}.wy-table-secondary{color:grey;font-size:90%}.wy-table-tertiary{color:grey;font-size:80%}.rst-content table.docutils:not(.field-list) tr:nth-child(2n-1) td,.wy-table-backed,.wy-table-odd td,.wy-table-striped tr:nth-child(2n-1) td{background-color:#f3f6f6}.rst-content table.docutils,.wy-table-bordered-all{border:1px solid #e1e4e5}.rst-content table.docutils td,.wy-table-bordered-all td{border-bottom:1px solid #e1e4e5;border-left:1px solid #e1e4e5}.rst-content table.docutils tbody>tr:last-child td,.wy-table-bordered-all tbody>tr:last-child td{border-bottom-width:0}.wy-table-bordered{border:1px solid #e1e4e5}.wy-table-bordered-rows td{border-bottom:1px solid #e1e4e5}.wy-table-bordered-rows tbody>tr:last-child td{border-bottom-width:0}.wy-table-horizontal td,.wy-table-horizontal th{border-width:0 0 1px;border-bottom:1px solid #e1e4e5}.wy-table-horizontal tbody>tr:last-child td{border-bottom-width:0}.wy-table-responsive{margin-bottom:24px;max-width:100%;overflow:auto}.wy-table-responsive table{margin-bottom:0!important}.wy-table-responsive table td,.wy-table-responsive table th{white-space:nowrap}a{color:#2980b9;text-decoration:none;cursor:pointer}a:hover{color:#3091d1}a:visited{color:#9b59b6}html{height:100%}body,html{overflow-x:hidden}body{font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;font-weight:400;color:#404040;min-height:100%;background:#edf0f2}.wy-text-left{text-align:left}.wy-text-center{text-align:center}.wy-text-right{text-align:right}.wy-text-large{font-size:120%}.wy-text-normal{font-size:100%}.wy-text-small,small{font-size:80%}.wy-text-strike{text-decoration:line-through}.wy-text-warning{color:#e67e22!important}a.wy-text-warning:hover{color:#eb9950!important}.wy-text-info{color:#2980b9!important}a.wy-text-info:hover{color:#409ad5!important}.wy-text-success{color:#27ae60!important}a.wy-text-success:hover{color:#36d278!important}.wy-text-danger{color:#e74c3c!important}a.wy-text-danger:hover{color:#ed7669!important}.wy-text-neutral{color:#404040!important}a.wy-text-neutral:hover{color:#595959!important}.rst-content .toctree-wrapper>p.caption,h1,h2,h3,h4,h5,h6,legend{margin-top:0;font-weight:700;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif}p{line-height:24px;font-size:16px;margin:0 0 24px}h1{font-size:175%}.rst-content .toctree-wrapper>p.caption,h2{font-size:150%}h3{font-size:125%}h4{font-size:115%}h5{font-size:110%}h6{font-size:100%}hr{display:block;height:1px;border:0;border-top:1px solid #e1e4e5;margin:24px 0;padding:0}.rst-content code,.rst-content tt,code{white-space:nowrap;max-width:100%;background:#fff;border:1px solid #e1e4e5;font-size:75%;padding:0 5px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;color:#e74c3c;overflow-x:auto}.rst-content tt.code-large,code.code-large{font-size:90%}.rst-content .section ul,.rst-content .toctree-wrapper ul,.rst-content section ul,.wy-plain-list-disc,article ul{list-style:disc;line-height:24px;margin-bottom:24px}.rst-content .section ul li,.rst-content .toctree-wrapper ul li,.rst-content section ul li,.wy-plain-list-disc li,article ul li{list-style:disc;margin-left:24px}.rst-content .section ul li p:last-child,.rst-content .section ul li ul,.rst-content .toctree-wrapper ul li p:last-child,.rst-content .toctree-wrapper ul li ul,.rst-content section ul li p:last-child,.rst-content section ul li ul,.wy-plain-list-disc li p:last-child,.wy-plain-list-disc li ul,article ul li p:last-child,article ul li ul{margin-bottom:0}.rst-content .section ul li li,.rst-content .toctree-wrapper ul li li,.rst-content section ul li li,.wy-plain-list-disc li li,article ul li li{list-style:circle}.rst-content .section ul li li li,.rst-content .toctree-wrapper ul li li li,.rst-content section ul li li li,.wy-plain-list-disc li li li,article ul li li li{list-style:square}.rst-content .section ul li ol li,.rst-content .toctree-wrapper ul li ol li,.rst-content section ul li ol li,.wy-plain-list-disc li ol li,article ul li ol li{list-style:decimal}.rst-content .section ol,.rst-content .section ol.arabic,.rst-content .toctree-wrapper ol,.rst-content .toctree-wrapper ol.arabic,.rst-content section ol,.rst-content section ol.arabic,.wy-plain-list-decimal,article ol{list-style:decimal;line-height:24px;margin-bottom:24px}.rst-content .section ol.arabic li,.rst-content .section ol li,.rst-content .toctree-wrapper ol.arabic li,.rst-content .toctree-wrapper ol li,.rst-content section ol.arabic li,.rst-content section ol li,.wy-plain-list-decimal li,article ol li{list-style:decimal;margin-left:24px}.rst-content .section ol.arabic li ul,.rst-content .section ol li p:last-child,.rst-content .section ol li ul,.rst-content .toctree-wrapper ol.arabic li ul,.rst-content .toctree-wrapper ol li p:last-child,.rst-content .toctree-wrapper ol li ul,.rst-content section ol.arabic li ul,.rst-content section ol li p:last-child,.rst-content section ol li ul,.wy-plain-list-decimal li p:last-child,.wy-plain-list-decimal li ul,article ol li p:last-child,article ol li ul{margin-bottom:0}.rst-content .section ol.arabic li ul li,.rst-content .section ol li ul li,.rst-content .toctree-wrapper ol.arabic li ul li,.rst-content .toctree-wrapper ol li ul li,.rst-content section ol.arabic li ul li,.rst-content section ol li ul li,.wy-plain-list-decimal li ul li,article ol li ul li{list-style:disc}.wy-breadcrumbs{*zoom:1}.wy-breadcrumbs:after,.wy-breadcrumbs:before{display:table;content:""}.wy-breadcrumbs:after{clear:both}.wy-breadcrumbs>li{display:inline-block;padding-top:5px}.wy-breadcrumbs>li.wy-breadcrumbs-aside{float:right}.rst-content .wy-breadcrumbs>li code,.rst-content .wy-breadcrumbs>li tt,.wy-breadcrumbs>li .rst-content tt,.wy-breadcrumbs>li code{all:inherit;color:inherit}.breadcrumb-item:before{content:"/";color:#bbb;font-size:13px;padding:0 6px 0 3px}.wy-breadcrumbs-extra{margin-bottom:0;color:#b3b3b3;font-size:80%;display:inline-block}@media screen and (max-width:480px){.wy-breadcrumbs-extra,.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}@media print{.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}html{font-size:16px}.wy-affix{position:fixed;top:1.618em}.wy-menu a:hover{text-decoration:none}.wy-menu-horiz{*zoom:1}.wy-menu-horiz:after,.wy-menu-horiz:before{display:table;content:""}.wy-menu-horiz:after{clear:both}.wy-menu-horiz li,.wy-menu-horiz ul{display:inline-block}.wy-menu-horiz li:hover{background:hsla(0,0%,100%,.1)}.wy-menu-horiz li.divide-left{border-left:1px solid #404040}.wy-menu-horiz li.divide-right{border-right:1px solid #404040}.wy-menu-horiz a{height:32px;display:inline-block;line-height:32px;padding:0 16px}.wy-menu-vertical{width:300px}.wy-menu-vertical header,.wy-menu-vertical p.caption{color:#55a5d9;height:32px;line-height:32px;padding:0 1.618em;margin:12px 0 0;display:block;font-weight:700;text-transform:uppercase;font-size:85%;white-space:nowrap}.wy-menu-vertical ul{margin-bottom:0}.wy-menu-vertical li.divide-top{border-top:1px solid #404040}.wy-menu-vertical li.divide-bottom{border-bottom:1px solid #404040}.wy-menu-vertical li.current{background:#e3e3e3}.wy-menu-vertical li.current a{color:grey;border-right:1px solid #c9c9c9;padding:.4045em 2.427em}.wy-menu-vertical li.current a:hover{background:#d6d6d6}.rst-content .wy-menu-vertical li tt,.wy-menu-vertical li .rst-content tt,.wy-menu-vertical li code{border:none;background:inherit;color:inherit;padding-left:0;padding-right:0}.wy-menu-vertical li button.toctree-expand{display:block;float:left;margin-left:-1.2em;line-height:18px;color:#4d4d4d;border:none;background:none;padding:0}.wy-menu-vertical li.current>a,.wy-menu-vertical li.on a{color:#404040;font-weight:700;position:relative;background:#fcfcfc;border:none;padding:.4045em 1.618em}.wy-menu-vertical li.current>a:hover,.wy-menu-vertical li.on a:hover{background:#fcfcfc}.wy-menu-vertical li.current>a:hover button.toctree-expand,.wy-menu-vertical li.on a:hover button.toctree-expand{color:grey}.wy-menu-vertical li.current>a button.toctree-expand,.wy-menu-vertical li.on a button.toctree-expand{display:block;line-height:18px;color:#333}.wy-menu-vertical li.toctree-l1.current>a{border-bottom:1px solid #c9c9c9;border-top:1px solid #c9c9c9}.wy-menu-vertical .toctree-l1.current .toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .toctree-l11>ul{display:none}.wy-menu-vertical .toctree-l1.current .current.toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .current.toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .current.toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .current.toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .current.toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .current.toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .current.toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .current.toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .current.toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .current.toctree-l11>ul{display:block}.wy-menu-vertical li.toctree-l3,.wy-menu-vertical li.toctree-l4{font-size:.9em}.wy-menu-vertical li.toctree-l2 a,.wy-menu-vertical li.toctree-l3 a,.wy-menu-vertical li.toctree-l4 a,.wy-menu-vertical li.toctree-l5 a,.wy-menu-vertical li.toctree-l6 a,.wy-menu-vertical li.toctree-l7 a,.wy-menu-vertical li.toctree-l8 a,.wy-menu-vertical li.toctree-l9 a,.wy-menu-vertical li.toctree-l10 a{color:#404040}.wy-menu-vertical li.toctree-l2 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l3 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l4 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l5 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l6 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l7 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l8 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l9 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l10 a:hover button.toctree-expand{color:grey}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{display:block}.wy-menu-vertical li.toctree-l2.current>a{padding:.4045em 2.427em}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{padding:.4045em 1.618em .4045em 4.045em}.wy-menu-vertical li.toctree-l3.current>a{padding:.4045em 4.045em}.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{padding:.4045em 1.618em .4045em 5.663em}.wy-menu-vertical li.toctree-l4.current>a{padding:.4045em 5.663em}.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a{padding:.4045em 1.618em .4045em 7.281em}.wy-menu-vertical li.toctree-l5.current>a{padding:.4045em 7.281em}.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a{padding:.4045em 1.618em .4045em 8.899em}.wy-menu-vertical li.toctree-l6.current>a{padding:.4045em 8.899em}.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a{padding:.4045em 1.618em .4045em 10.517em}.wy-menu-vertical li.toctree-l7.current>a{padding:.4045em 10.517em}.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a{padding:.4045em 1.618em .4045em 12.135em}.wy-menu-vertical li.toctree-l8.current>a{padding:.4045em 12.135em}.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a{padding:.4045em 1.618em .4045em 13.753em}.wy-menu-vertical li.toctree-l9.current>a{padding:.4045em 13.753em}.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a{padding:.4045em 1.618em .4045em 15.371em}.wy-menu-vertical li.toctree-l10.current>a{padding:.4045em 15.371em}.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{padding:.4045em 1.618em .4045em 16.989em}.wy-menu-vertical li.toctree-l2.current>a,.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{background:#c9c9c9}.wy-menu-vertical li.toctree-l2 button.toctree-expand{color:#a3a3a3}.wy-menu-vertical li.toctree-l3.current>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{background:#bdbdbd}.wy-menu-vertical li.toctree-l3 button.toctree-expand{color:#969696}.wy-menu-vertical li.current ul{display:block}.wy-menu-vertical li ul{margin-bottom:0;display:none}.wy-menu-vertical li ul li a{margin-bottom:0;color:#d9d9d9;font-weight:400}.wy-menu-vertical a{line-height:18px;padding:.4045em 1.618em;display:block;position:relative;font-size:90%;color:#d9d9d9}.wy-menu-vertical a:hover{background-color:#4e4a4a;cursor:pointer}.wy-menu-vertical a:hover button.toctree-expand{color:#d9d9d9}.wy-menu-vertical a:active{background-color:#2980b9;cursor:pointer;color:#fff}.wy-menu-vertical a:active button.toctree-expand{color:#fff}.wy-side-nav-search{display:block;width:300px;padding:.809em;margin-bottom:.809em;z-index:200;background-color:#2980b9;text-align:center;color:#fcfcfc}.wy-side-nav-search input[type=text]{width:100%;border-radius:50px;padding:6px 12px;border-color:#2472a4}.wy-side-nav-search img{display:block;margin:auto auto .809em;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a{color:#fcfcfc;font-size:100%;font-weight:700;display:inline-block;padding:4px 6px;margin-bottom:.809em;max-width:100%}.wy-side-nav-search .wy-dropdown>a:hover,.wy-side-nav-search .wy-dropdown>aactive,.wy-side-nav-search .wy-dropdown>afocus,.wy-side-nav-search>a:hover,.wy-side-nav-search>aactive,.wy-side-nav-search>afocus{background:hsla(0,0%,100%,.1)}.wy-side-nav-search .wy-dropdown>a img.logo,.wy-side-nav-search>a img.logo{display:block;margin:0 auto;height:auto;width:auto;border-radius:0;max-width:100%;background:transparent}.wy-side-nav-search .wy-dropdown>a.icon,.wy-side-nav-search>a.icon{display:block}.wy-side-nav-search .wy-dropdown>a.icon img.logo,.wy-side-nav-search>a.icon img.logo{margin-top:.85em}.wy-side-nav-search>div.switch-menus{position:relative;display:block;margin-top:-.4045em;margin-bottom:.809em;font-weight:400;color:hsla(0,0%,100%,.3)}.wy-side-nav-search>div.switch-menus>div.language-switch,.wy-side-nav-search>div.switch-menus>div.version-switch{display:inline-block;padding:.2em}.wy-side-nav-search>div.switch-menus>div.language-switch select,.wy-side-nav-search>div.switch-menus>div.version-switch select{display:inline-block;margin-right:-2rem;padding-right:2rem;max-width:240px;text-align-last:center;background:none;border:none;border-radius:0;box-shadow:none;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;font-size:1em;font-weight:400;color:hsla(0,0%,100%,.3);cursor:pointer;appearance:none;-webkit-appearance:none;-moz-appearance:none}.wy-side-nav-search>div.switch-menus>div.language-switch select:active,.wy-side-nav-search>div.switch-menus>div.language-switch select:focus,.wy-side-nav-search>div.switch-menus>div.language-switch select:hover,.wy-side-nav-search>div.switch-menus>div.version-switch select:active,.wy-side-nav-search>div.switch-menus>div.version-switch select:focus,.wy-side-nav-search>div.switch-menus>div.version-switch select:hover{background:hsla(0,0%,100%,.1);color:hsla(0,0%,100%,.5)}.wy-side-nav-search>div.switch-menus>div.language-switch select option,.wy-side-nav-search>div.switch-menus>div.version-switch select option{color:#000}.wy-side-nav-search>div.switch-menus>div.language-switch:has(>select):after,.wy-side-nav-search>div.switch-menus>div.version-switch:has(>select):after{display:inline-block;width:1.5em;height:100%;padding:.1em;content:"\f0d7";font-size:1em;line-height:1.2em;font-family:FontAwesome;text-align:center;pointer-events:none;box-sizing:border-box}.wy-nav .wy-menu-vertical header{color:#2980b9}.wy-nav .wy-menu-vertical a{color:#b3b3b3}.wy-nav .wy-menu-vertical a:hover{background-color:#2980b9;color:#fff}[data-menu-wrap]{-webkit-transition:all .2s ease-in;-moz-transition:all .2s ease-in;transition:all .2s ease-in;position:absolute;opacity:1;width:100%;opacity:0}[data-menu-wrap].move-center{left:0;right:auto;opacity:1}[data-menu-wrap].move-left{right:auto;left:-100%;opacity:0}[data-menu-wrap].move-right{right:-100%;left:auto;opacity:0}.wy-body-for-nav{background:#fcfcfc}.wy-grid-for-nav{position:absolute;width:100%;height:100%}.wy-nav-side{position:fixed;top:0;bottom:0;left:0;padding-bottom:2em;width:300px;overflow-x:hidden;overflow-y:hidden;min-height:100%;color:#9b9b9b;background:#343131;z-index:200}.wy-side-scroll{width:320px;position:relative;overflow-x:hidden;overflow-y:scroll;height:100%}.wy-nav-top{display:none;background:#2980b9;color:#fff;padding:.4045em .809em;position:relative;line-height:50px;text-align:center;font-size:100%;*zoom:1}.wy-nav-top:after,.wy-nav-top:before{display:table;content:""}.wy-nav-top:after{clear:both}.wy-nav-top a{color:#fff;font-weight:700}.wy-nav-top img{margin-right:12px;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-nav-top i{font-size:30px;float:left;cursor:pointer;padding-top:inherit}.wy-nav-content-wrap{margin-left:300px;background:#fcfcfc;min-height:100%}.wy-nav-content{padding:1.618em 3.236em;height:100%;max-width:800px;margin:auto}.wy-body-mask{position:fixed;width:100%;height:100%;background:rgba(0,0,0,.2);display:none;z-index:499}.wy-body-mask.on{display:block}footer{color:grey}footer p{margin-bottom:12px}.rst-content footer span.commit tt,footer span.commit .rst-content tt,footer span.commit code{padding:0;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:1em;background:none;border:none;color:grey}.rst-footer-buttons{*zoom:1}.rst-footer-buttons:after,.rst-footer-buttons:before{width:100%;display:table;content:""}.rst-footer-buttons:after{clear:both}.rst-breadcrumbs-buttons{margin-top:12px;*zoom:1}.rst-breadcrumbs-buttons:after,.rst-breadcrumbs-buttons:before{display:table;content:""}.rst-breadcrumbs-buttons:after{clear:both}#search-results .search li{margin-bottom:24px;border-bottom:1px solid #e1e4e5;padding-bottom:24px}#search-results .search li:first-child{border-top:1px solid #e1e4e5;padding-top:24px}#search-results .search li a{font-size:120%;margin-bottom:12px;display:inline-block}#search-results .context{color:grey;font-size:90%}.genindextable li>ul{margin-left:24px}@media screen and (max-width:768px){.wy-body-for-nav{background:#fcfcfc}.wy-nav-top{display:block}.wy-nav-side{left:-300px}.wy-nav-side.shift{width:85%;left:0}.wy-menu.wy-menu-vertical,.wy-side-nav-search,.wy-side-scroll{width:auto}.wy-nav-content-wrap{margin-left:0}.wy-nav-content-wrap .wy-nav-content{padding:1.618em}.wy-nav-content-wrap.shift{position:fixed;min-width:100%;left:85%;top:0;height:100%;overflow:hidden}}@media screen and (min-width:1100px){.wy-nav-content-wrap{background:rgba(0,0,0,.05)}.wy-nav-content{margin:0;background:#fcfcfc}}@media print{.rst-versions,.wy-nav-side,footer{display:none}.wy-nav-content-wrap{margin-left:0}}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60;*zoom:1}.rst-versions .rst-current-version:after,.rst-versions .rst-current-version:before{display:table;content:""}.rst-versions .rst-current-version:after{clear:both}.rst-content .code-block-caption .rst-versions .rst-current-version .headerlink,.rst-content .eqno .rst-versions .rst-current-version .headerlink,.rst-content .rst-versions .rst-current-version .admonition-title,.rst-content code.download .rst-versions .rst-current-version span:first-child,.rst-content dl dt .rst-versions .rst-current-version .headerlink,.rst-content h1 .rst-versions .rst-current-version .headerlink,.rst-content h2 .rst-versions .rst-current-version .headerlink,.rst-content h3 .rst-versions .rst-current-version .headerlink,.rst-content h4 .rst-versions .rst-current-version .headerlink,.rst-content h5 .rst-versions .rst-current-version .headerlink,.rst-content h6 .rst-versions .rst-current-version .headerlink,.rst-content p .rst-versions .rst-current-version .headerlink,.rst-content table>caption .rst-versions .rst-current-version .headerlink,.rst-content tt.download .rst-versions .rst-current-version span:first-child,.rst-versions .rst-current-version .fa,.rst-versions .rst-current-version .icon,.rst-versions .rst-current-version .rst-content .admonition-title,.rst-versions .rst-current-version .rst-content .code-block-caption .headerlink,.rst-versions .rst-current-version .rst-content .eqno .headerlink,.rst-versions .rst-current-version .rst-content code.download span:first-child,.rst-versions .rst-current-version .rst-content dl dt .headerlink,.rst-versions .rst-current-version .rst-content h1 .headerlink,.rst-versions .rst-current-version .rst-content h2 .headerlink,.rst-versions .rst-current-version .rst-content h3 .headerlink,.rst-versions .rst-current-version .rst-content h4 .headerlink,.rst-versions .rst-current-version .rst-content h5 .headerlink,.rst-versions .rst-current-version .rst-content h6 .headerlink,.rst-versions .rst-current-version .rst-content p .headerlink,.rst-versions .rst-current-version .rst-content table>caption .headerlink,.rst-versions .rst-current-version .rst-content tt.download span:first-child,.rst-versions .rst-current-version .wy-menu-vertical li button.toctree-expand,.wy-menu-vertical li .rst-versions .rst-current-version button.toctree-expand{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions .rst-other-versions .rtd-current-item{font-weight:700}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}}#flyout-search-form{padding:6px}.rst-content .toctree-wrapper>p.caption,.rst-content h1,.rst-content h2,.rst-content h3,.rst-content h4,.rst-content h5,.rst-content h6{margin-bottom:24px}.rst-content img{max-width:100%;height:auto}.rst-content div.figure,.rst-content figure{margin-bottom:24px}.rst-content div.figure .caption-text,.rst-content figure .caption-text{font-style:italic}.rst-content div.figure p:last-child.caption,.rst-content figure p:last-child.caption{margin-bottom:0}.rst-content div.figure.align-center,.rst-content figure.align-center{text-align:center}.rst-content .section>a>img,.rst-content .section>img,.rst-content section>a>img,.rst-content section>img{margin-bottom:24px}.rst-content abbr[title]{text-decoration:none}.rst-content.style-external-links a.reference.external:after{font-family:FontAwesome;content:"\f08e";color:#b3b3b3;vertical-align:super;font-size:60%;margin:0 .2em}.rst-content blockquote{margin-left:24px;line-height:24px;margin-bottom:24px}.rst-content pre.literal-block{white-space:pre;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;display:block;overflow:auto}.rst-content div[class^=highlight],.rst-content pre.literal-block{border:1px solid #e1e4e5;overflow-x:auto;margin:1px 0 24px}.rst-content div[class^=highlight] div[class^=highlight],.rst-content pre.literal-block div[class^=highlight]{padding:0;border:none;margin:0}.rst-content div[class^=highlight] td.code{width:100%}.rst-content .linenodiv pre{border-right:1px solid #e6e9ea;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;user-select:none;pointer-events:none}.rst-content div[class^=highlight] pre{white-space:pre;margin:0;padding:12px;display:block;overflow:auto}.rst-content div[class^=highlight] pre .hll{display:block;margin:0 -12px;padding:0 12px}.rst-content .linenodiv pre,.rst-content div[class^=highlight] pre,.rst-content pre.literal-block{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:12px;line-height:1.4}.rst-content div.highlight .gp,.rst-content div.highlight span.linenos{user-select:none;pointer-events:none}.rst-content div.highlight span.linenos{display:inline-block;padding-left:0;padding-right:12px;margin-right:12px;border-right:1px solid #e6e9ea}.rst-content .code-block-caption{font-style:italic;font-size:85%;line-height:1;padding:1em 0;text-align:center}@media print{.rst-content .codeblock,.rst-content div[class^=highlight],.rst-content div[class^=highlight] pre{white-space:pre-wrap}}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning{clear:both}.rst-content .admonition-todo .last,.rst-content .admonition-todo>:last-child,.rst-content .admonition .last,.rst-content .admonition>:last-child,.rst-content .attention .last,.rst-content .attention>:last-child,.rst-content .caution .last,.rst-content .caution>:last-child,.rst-content .danger .last,.rst-content .danger>:last-child,.rst-content .error .last,.rst-content .error>:last-child,.rst-content .hint .last,.rst-content .hint>:last-child,.rst-content .important .last,.rst-content .important>:last-child,.rst-content .note .last,.rst-content .note>:last-child,.rst-content .seealso .last,.rst-content .seealso>:last-child,.rst-content .tip .last,.rst-content .tip>:last-child,.rst-content .warning .last,.rst-content .warning>:last-child{margin-bottom:0}.rst-content .admonition-title:before{margin-right:4px}.rst-content .admonition table{border-color:rgba(0,0,0,.1)}.rst-content .admonition table td,.rst-content .admonition table th{background:transparent!important;border-color:rgba(0,0,0,.1)!important}.rst-content .section ol.loweralpha,.rst-content .section ol.loweralpha>li,.rst-content .toctree-wrapper ol.loweralpha,.rst-content .toctree-wrapper ol.loweralpha>li,.rst-content section ol.loweralpha,.rst-content section ol.loweralpha>li{list-style:lower-alpha}.rst-content .section ol.upperalpha,.rst-content .section ol.upperalpha>li,.rst-content .toctree-wrapper ol.upperalpha,.rst-content .toctree-wrapper ol.upperalpha>li,.rst-content section ol.upperalpha,.rst-content section ol.upperalpha>li{list-style:upper-alpha}.rst-content .section ol li>*,.rst-content .section ul li>*,.rst-content .toctree-wrapper ol li>*,.rst-content .toctree-wrapper ul li>*,.rst-content section ol li>*,.rst-content section ul li>*{margin-top:12px;margin-bottom:12px}.rst-content .section ol li>:first-child,.rst-content .section ul li>:first-child,.rst-content .toctree-wrapper ol li>:first-child,.rst-content .toctree-wrapper ul li>:first-child,.rst-content section ol li>:first-child,.rst-content section ul li>:first-child{margin-top:0}.rst-content .section ol li>p,.rst-content .section ol li>p:last-child,.rst-content .section ul li>p,.rst-content .section ul li>p:last-child,.rst-content .toctree-wrapper ol li>p,.rst-content .toctree-wrapper ol li>p:last-child,.rst-content .toctree-wrapper ul li>p,.rst-content .toctree-wrapper ul li>p:last-child,.rst-content section ol li>p,.rst-content section ol li>p:last-child,.rst-content section ul li>p,.rst-content section ul li>p:last-child{margin-bottom:12px}.rst-content .section ol li>p:only-child,.rst-content .section ol li>p:only-child:last-child,.rst-content .section ul li>p:only-child,.rst-content .section ul li>p:only-child:last-child,.rst-content .toctree-wrapper ol li>p:only-child,.rst-content .toctree-wrapper ol li>p:only-child:last-child,.rst-content .toctree-wrapper ul li>p:only-child,.rst-content .toctree-wrapper ul li>p:only-child:last-child,.rst-content section ol li>p:only-child,.rst-content section ol li>p:only-child:last-child,.rst-content section ul li>p:only-child,.rst-content section ul li>p:only-child:last-child{margin-bottom:0}.rst-content .section ol li>ol,.rst-content .section ol li>ul,.rst-content .section ul li>ol,.rst-content .section ul li>ul,.rst-content .toctree-wrapper ol li>ol,.rst-content .toctree-wrapper ol li>ul,.rst-content .toctree-wrapper ul li>ol,.rst-content .toctree-wrapper ul li>ul,.rst-content section ol li>ol,.rst-content section ol li>ul,.rst-content section ul li>ol,.rst-content section ul li>ul{margin-bottom:12px}.rst-content .section ol.simple li>*,.rst-content .section ol.simple li ol,.rst-content .section ol.simple li ul,.rst-content .section ul.simple li>*,.rst-content .section ul.simple li ol,.rst-content .section ul.simple li ul,.rst-content .toctree-wrapper ol.simple li>*,.rst-content .toctree-wrapper ol.simple li ol,.rst-content .toctree-wrapper ol.simple li ul,.rst-content .toctree-wrapper ul.simple li>*,.rst-content .toctree-wrapper ul.simple li ol,.rst-content .toctree-wrapper ul.simple li ul,.rst-content section ol.simple li>*,.rst-content section ol.simple li ol,.rst-content section ol.simple li ul,.rst-content section ul.simple li>*,.rst-content section ul.simple li ol,.rst-content section ul.simple li ul{margin-top:0;margin-bottom:0}.rst-content .line-block{margin-left:0;margin-bottom:24px;line-height:24px}.rst-content .line-block .line-block{margin-left:24px;margin-bottom:0}.rst-content .topic-title{font-weight:700;margin-bottom:12px}.rst-content .toc-backref{color:#404040}.rst-content .align-right{float:right;margin:0 0 24px 24px}.rst-content .align-left{float:left;margin:0 24px 24px 0}.rst-content .align-center{margin:auto}.rst-content .align-center:not(table){display:block}.rst-content .code-block-caption .headerlink,.rst-content .eqno .headerlink,.rst-content .toctree-wrapper>p.caption .headerlink,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content p .headerlink,.rst-content table>caption .headerlink{opacity:0;font-size:14px;font-family:FontAwesome;margin-left:.5em}.rst-content .code-block-caption .headerlink:focus,.rst-content .code-block-caption:hover .headerlink,.rst-content .eqno .headerlink:focus,.rst-content .eqno:hover .headerlink,.rst-content .toctree-wrapper>p.caption .headerlink:focus,.rst-content .toctree-wrapper>p.caption:hover .headerlink,.rst-content dl dt .headerlink:focus,.rst-content dl dt:hover .headerlink,.rst-content h1 .headerlink:focus,.rst-content h1:hover .headerlink,.rst-content h2 .headerlink:focus,.rst-content h2:hover .headerlink,.rst-content h3 .headerlink:focus,.rst-content h3:hover .headerlink,.rst-content h4 .headerlink:focus,.rst-content h4:hover .headerlink,.rst-content h5 .headerlink:focus,.rst-content h5:hover .headerlink,.rst-content h6 .headerlink:focus,.rst-content h6:hover .headerlink,.rst-content p.caption .headerlink:focus,.rst-content p.caption:hover .headerlink,.rst-content p .headerlink:focus,.rst-content p:hover .headerlink,.rst-content table>caption .headerlink:focus,.rst-content table>caption:hover .headerlink{opacity:1}.rst-content p a{overflow-wrap:anywhere}.rst-content .wy-table td p,.rst-content .wy-table td ul,.rst-content .wy-table th p,.rst-content .wy-table th ul,.rst-content table.docutils td p,.rst-content table.docutils td ul,.rst-content table.docutils th p,.rst-content table.docutils th ul,.rst-content table.field-list td p,.rst-content table.field-list td ul,.rst-content table.field-list th p,.rst-content table.field-list th ul{font-size:inherit}.rst-content .btn:focus{outline:2px solid}.rst-content table>caption .headerlink:after{font-size:12px}.rst-content .centered{text-align:center}.rst-content .sidebar{float:right;width:40%;display:block;margin:0 0 24px 24px;padding:24px;background:#f3f6f6;border:1px solid #e1e4e5}.rst-content .sidebar dl,.rst-content .sidebar p,.rst-content .sidebar ul{font-size:90%}.rst-content .sidebar .last,.rst-content .sidebar>:last-child{margin-bottom:0}.rst-content .sidebar .sidebar-title{display:block;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif;font-weight:700;background:#e1e4e5;padding:6px 12px;margin:-24px -24px 24px;font-size:100%}.rst-content .highlighted{background:#f1c40f;box-shadow:0 0 0 2px #f1c40f;display:inline;font-weight:700}.rst-content .citation-reference,.rst-content .footnote-reference{vertical-align:baseline;position:relative;top:-.4em;line-height:0;font-size:90%}.rst-content .citation-reference>span.fn-bracket,.rst-content .footnote-reference>span.fn-bracket{display:none}.rst-content .hlist{width:100%}.rst-content dl dt span.classifier:before{content:" : "}.rst-content dl dt span.classifier-delimiter{display:none!important}html.writer-html4 .rst-content table.docutils.citation,html.writer-html4 .rst-content table.docutils.footnote{background:none;border:none}html.writer-html4 .rst-content table.docutils.citation td,html.writer-html4 .rst-content table.docutils.citation tr,html.writer-html4 .rst-content table.docutils.footnote td,html.writer-html4 .rst-content table.docutils.footnote tr{border:none;background-color:transparent!important;white-space:normal}html.writer-html4 .rst-content table.docutils.citation td.label,html.writer-html4 .rst-content table.docutils.footnote td.label{padding-left:0;padding-right:0;vertical-align:top}html.writer-html5 .rst-content dl.citation,html.writer-html5 .rst-content dl.field-list,html.writer-html5 .rst-content dl.footnote{display:grid;grid-template-columns:auto minmax(80%,95%)}html.writer-html5 .rst-content dl.citation>dt,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dt{display:inline-grid;grid-template-columns:max-content auto}html.writer-html5 .rst-content aside.citation,html.writer-html5 .rst-content aside.footnote,html.writer-html5 .rst-content div.citation{display:grid;grid-template-columns:auto auto minmax(.65rem,auto) minmax(40%,95%)}html.writer-html5 .rst-content aside.citation>span.label,html.writer-html5 .rst-content aside.footnote>span.label,html.writer-html5 .rst-content div.citation>span.label{grid-column-start:1;grid-column-end:2}html.writer-html5 .rst-content aside.citation>span.backrefs,html.writer-html5 .rst-content aside.footnote>span.backrefs,html.writer-html5 .rst-content div.citation>span.backrefs{grid-column-start:2;grid-column-end:3;grid-row-start:1;grid-row-end:3}html.writer-html5 .rst-content aside.citation>p,html.writer-html5 .rst-content aside.footnote>p,html.writer-html5 .rst-content div.citation>p{grid-column-start:4;grid-column-end:5}html.writer-html5 .rst-content dl.citation,html.writer-html5 .rst-content dl.field-list,html.writer-html5 .rst-content dl.footnote{margin-bottom:24px}html.writer-html5 .rst-content dl.citation>dt,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dt{padding-left:1rem}html.writer-html5 .rst-content dl.citation>dd,html.writer-html5 .rst-content dl.citation>dt,html.writer-html5 .rst-content dl.field-list>dd,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dd,html.writer-html5 .rst-content dl.footnote>dt{margin-bottom:0}html.writer-html5 .rst-content dl.citation,html.writer-html5 .rst-content dl.footnote{font-size:.9rem}html.writer-html5 .rst-content dl.citation>dt,html.writer-html5 .rst-content dl.footnote>dt{margin:0 .5rem .5rem 0;line-height:1.2rem;word-break:break-all;font-weight:400}html.writer-html5 .rst-content dl.citation>dt>span.brackets:before,html.writer-html5 .rst-content dl.footnote>dt>span.brackets:before{content:"["}html.writer-html5 .rst-content dl.citation>dt>span.brackets:after,html.writer-html5 .rst-content dl.footnote>dt>span.brackets:after{content:"]"}html.writer-html5 .rst-content dl.citation>dt>span.fn-backref,html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref{text-align:left;font-style:italic;margin-left:.65rem;word-break:break-word;word-spacing:-.1rem;max-width:5rem}html.writer-html5 .rst-content dl.citation>dt>span.fn-backref>a,html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref>a{word-break:keep-all}html.writer-html5 .rst-content dl.citation>dt>span.fn-backref>a:not(:first-child):before,html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref>a:not(:first-child):before{content:" "}html.writer-html5 .rst-content dl.citation>dd,html.writer-html5 .rst-content dl.footnote>dd{margin:0 0 .5rem;line-height:1.2rem}html.writer-html5 .rst-content dl.citation>dd p,html.writer-html5 .rst-content dl.footnote>dd p{font-size:.9rem}html.writer-html5 .rst-content aside.citation,html.writer-html5 .rst-content aside.footnote,html.writer-html5 .rst-content div.citation{padding-left:1rem;padding-right:1rem;font-size:.9rem;line-height:1.2rem}html.writer-html5 .rst-content aside.citation p,html.writer-html5 .rst-content aside.footnote p,html.writer-html5 .rst-content div.citation p{font-size:.9rem;line-height:1.2rem;margin-bottom:12px}html.writer-html5 .rst-content aside.citation span.backrefs,html.writer-html5 .rst-content aside.footnote span.backrefs,html.writer-html5 .rst-content div.citation span.backrefs{text-align:left;font-style:italic;margin-left:.65rem;word-break:break-word;word-spacing:-.1rem;max-width:5rem}html.writer-html5 .rst-content aside.citation span.backrefs>a,html.writer-html5 .rst-content aside.footnote span.backrefs>a,html.writer-html5 .rst-content div.citation span.backrefs>a{word-break:keep-all}html.writer-html5 .rst-content aside.citation span.backrefs>a:not(:first-child):before,html.writer-html5 .rst-content aside.footnote span.backrefs>a:not(:first-child):before,html.writer-html5 .rst-content div.citation span.backrefs>a:not(:first-child):before{content:" "}html.writer-html5 .rst-content aside.citation span.label,html.writer-html5 .rst-content aside.footnote span.label,html.writer-html5 .rst-content div.citation span.label{line-height:1.2rem}html.writer-html5 .rst-content aside.citation-list,html.writer-html5 .rst-content aside.footnote-list,html.writer-html5 .rst-content div.citation-list{margin-bottom:24px}html.writer-html5 .rst-content dl.option-list kbd{font-size:.9rem}.rst-content table.docutils.footnote,html.writer-html4 .rst-content table.docutils.citation,html.writer-html5 .rst-content aside.footnote,html.writer-html5 .rst-content aside.footnote-list aside.footnote,html.writer-html5 .rst-content div.citation-list>div.citation,html.writer-html5 .rst-content dl.citation,html.writer-html5 .rst-content dl.footnote{color:grey}.rst-content table.docutils.footnote code,.rst-content table.docutils.footnote tt,html.writer-html4 .rst-content table.docutils.citation code,html.writer-html4 .rst-content table.docutils.citation tt,html.writer-html5 .rst-content aside.footnote-list aside.footnote code,html.writer-html5 .rst-content aside.footnote-list aside.footnote tt,html.writer-html5 .rst-content aside.footnote code,html.writer-html5 .rst-content aside.footnote tt,html.writer-html5 .rst-content div.citation-list>div.citation code,html.writer-html5 .rst-content div.citation-list>div.citation tt,html.writer-html5 .rst-content dl.citation code,html.writer-html5 .rst-content dl.citation tt,html.writer-html5 .rst-content dl.footnote code,html.writer-html5 .rst-content dl.footnote tt{color:#555}.rst-content .wy-table-responsive.citation,.rst-content .wy-table-responsive.footnote{margin-bottom:0}.rst-content .wy-table-responsive.citation+:not(.citation),.rst-content .wy-table-responsive.footnote+:not(.footnote){margin-top:24px}.rst-content .wy-table-responsive.citation:last-child,.rst-content .wy-table-responsive.footnote:last-child{margin-bottom:24px}.rst-content table.docutils th{border-color:#e1e4e5}html.writer-html5 .rst-content table.docutils th{border:1px solid #e1e4e5}html.writer-html5 .rst-content table.docutils td>p,html.writer-html5 .rst-content table.docutils th>p{line-height:1rem;margin-bottom:0;font-size:.9rem}.rst-content table.docutils td .last,.rst-content table.docutils td .last>:last-child{margin-bottom:0}.rst-content table.field-list,.rst-content table.field-list td{border:none}.rst-content table.field-list td p{line-height:inherit}.rst-content table.field-list td>strong{display:inline-block}.rst-content table.field-list .field-name{padding-right:10px;text-align:left;white-space:nowrap}.rst-content table.field-list .field-body{text-align:left}.rst-content code,.rst-content tt{color:#000;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;padding:2px 5px}.rst-content code big,.rst-content code em,.rst-content tt big,.rst-content tt em{font-size:100%!important;line-height:normal}.rst-content code.literal,.rst-content tt.literal{color:#e74c3c;white-space:normal}.rst-content code.xref,.rst-content tt.xref,a .rst-content code,a .rst-content tt{font-weight:700;color:#404040;overflow-wrap:normal}.rst-content kbd,.rst-content pre,.rst-content samp{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace}.rst-content a code,.rst-content a tt{color:#2980b9}.rst-content dl{margin-bottom:24px}.rst-content dl dt{font-weight:700;margin-bottom:12px}.rst-content dl ol,.rst-content dl p,.rst-content dl table,.rst-content dl ul{margin-bottom:12px}.rst-content dl dd{margin:0 0 12px 24px;line-height:24px}.rst-content dl dd>ol:last-child,.rst-content dl dd>p:last-child,.rst-content dl dd>table:last-child,.rst-content dl dd>ul:last-child{margin-bottom:0}html.writer-html4 .rst-content dl:not(.docutils),html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple){margin-bottom:24px}html.writer-html4 .rst-content dl:not(.docutils)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt{display:table;margin:6px 0;font-size:90%;line-height:normal;background:#e7f2fa;color:#2980b9;border-top:3px solid #6ab0de;padding:6px;position:relative}html.writer-html4 .rst-content dl:not(.docutils)>dt:before,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt:before{color:#6ab0de}html.writer-html4 .rst-content dl:not(.docutils)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt{margin-bottom:6px;border:none;border-left:3px solid #ccc;background:#f0f0f0;color:#555}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils)>dt:first-child,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt:first-child{margin-top:0}html.writer-html4 .rst-content dl:not(.docutils) code.descclassname,html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descclassname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) code.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) tt.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) tt.descname{background-color:transparent;border:none;padding:0;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) tt.descname{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .optional,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .optional{display:inline-block;padding:0 4px;color:#000;font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .property,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .property{display:inline-block;padding-right:8px;max-width:100%}html.writer-html4 .rst-content dl:not(.docutils) .k,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .k{font-style:italic}html.writer-html4 .rst-content dl:not(.docutils) .descclassname,html.writer-html4 .rst-content dl:not(.docutils) .descname,html.writer-html4 .rst-content dl:not(.docutils) .sig-name,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .sig-name{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;color:#000}.rst-content .viewcode-back,.rst-content .viewcode-link{display:inline-block;color:#27ae60;font-size:80%;padding-left:24px}.rst-content .viewcode-back{display:block;float:right}.rst-content p.rubric{margin-bottom:12px;font-weight:700}.rst-content code.download,.rst-content tt.download{background:inherit;padding:inherit;font-weight:400;font-family:inherit;font-size:inherit;color:inherit;border:inherit;white-space:inherit}.rst-content code.download span:first-child,.rst-content tt.download span:first-child{-webkit-font-smoothing:subpixel-antialiased}.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{margin-right:4px}.rst-content .guilabel,.rst-content .menuselection{font-size:80%;font-weight:700;border-radius:4px;padding:2.4px 6px;margin:auto 2px}.rst-content .guilabel,.rst-content .menuselection{border:1px solid #7fbbe3;background:#e7f2fa}.rst-content :not(dl.option-list)>:not(dt):not(kbd):not(.kbd)>.kbd,.rst-content :not(dl.option-list)>:not(dt):not(kbd):not(.kbd)>kbd{color:inherit;font-size:80%;background-color:#fff;border:1px solid #a6a6a6;border-radius:4px;box-shadow:0 2px grey;padding:2.4px 6px;margin:auto 0}.rst-content .versionmodified{font-style:italic}@media screen and (max-width:480px){.rst-content .sidebar{width:100%}}span[id*=MathJax-Span]{color:#404040}.math{text-align:center}@font-face{font-family:Lato;src:url(fonts/lato-normal.woff2?bd03a2cc277bbbc338d464e679fe9942) format("woff2"),url(fonts/lato-normal.woff?27bd77b9162d388cb8d4c4217c7c5e2a) format("woff");font-weight:400;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold.woff2?cccb897485813c7c256901dbca54ecf2) format("woff2"),url(fonts/lato-bold.woff?d878b6c29b10beca227e9eef4246111b) format("woff");font-weight:700;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold-italic.woff2?0b6bb6725576b072c5d0b02ecdd1900d) format("woff2"),url(fonts/lato-bold-italic.woff?9c7e4e9eb485b4a121c760e61bc3707c) format("woff");font-weight:700;font-style:italic;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-normal-italic.woff2?4eb103b4d12be57cb1d040ed5e162e9d) format("woff2"),url(fonts/lato-normal-italic.woff?f28f2d6482446544ef1ea1ccc6dd5892) format("woff");font-weight:400;font-style:italic;font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:400;src:url(fonts/Roboto-Slab-Regular.woff2?7abf5b8d04d26a2cafea937019bca958) format("woff2"),url(fonts/Roboto-Slab-Regular.woff?c1be9284088d487c5e3ff0a10a92e58c) format("woff");font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:700;src:url(fonts/Roboto-Slab-Bold.woff2?9984f4a9bda09be08e83f2506954adbe) format("woff2"),url(fonts/Roboto-Slab-Bold.woff?bed5564a116b05148e3b3bea6fb1162a) format("woff");font-display:block} \ No newline at end of file diff --git a/_static/doctools.js b/_static/doctools.js new file mode 100644 index 000000000..0398ebb9f --- /dev/null +++ b/_static/doctools.js @@ -0,0 +1,149 @@ +/* + * Base JavaScript utilities for all Sphinx HTML documentation. + */ +"use strict"; + +const BLACKLISTED_KEY_CONTROL_ELEMENTS = new Set([ + "TEXTAREA", + "INPUT", + "SELECT", + "BUTTON", +]); + +const _ready = (callback) => { + if (document.readyState !== "loading") { + callback(); + } else { + document.addEventListener("DOMContentLoaded", callback); + } +}; + +/** + * Small JavaScript module for the documentation. + */ +const Documentation = { + init: () => { + Documentation.initDomainIndexTable(); + Documentation.initOnKeyListeners(); + }, + + /** + * i18n support + */ + TRANSLATIONS: {}, + PLURAL_EXPR: (n) => (n === 1 ? 0 : 1), + LOCALE: "unknown", + + // gettext and ngettext don't access this so that the functions + // can safely bound to a different name (_ = Documentation.gettext) + gettext: (string) => { + const translated = Documentation.TRANSLATIONS[string]; + switch (typeof translated) { + case "undefined": + return string; // no translation + case "string": + return translated; // translation exists + default: + return translated[0]; // (singular, plural) translation tuple exists + } + }, + + ngettext: (singular, plural, n) => { + const translated = Documentation.TRANSLATIONS[singular]; + if (typeof translated !== "undefined") + return translated[Documentation.PLURAL_EXPR(n)]; + return n === 1 ? singular : plural; + }, + + addTranslations: (catalog) => { + Object.assign(Documentation.TRANSLATIONS, catalog.messages); + Documentation.PLURAL_EXPR = new Function( + "n", + `return (${catalog.plural_expr})` + ); + Documentation.LOCALE = catalog.locale; + }, + + /** + * helper function to focus on search bar + */ + focusSearchBar: () => { + document.querySelectorAll("input[name=q]")[0]?.focus(); + }, + + /** + * Initialise the domain index toggle buttons + */ + initDomainIndexTable: () => { + const toggler = (el) => { + const idNumber = el.id.substr(7); + const toggledRows = document.querySelectorAll(`tr.cg-${idNumber}`); + if (el.src.substr(-9) === "minus.png") { + el.src = `${el.src.substr(0, el.src.length - 9)}plus.png`; + toggledRows.forEach((el) => (el.style.display = "none")); + } else { + el.src = `${el.src.substr(0, el.src.length - 8)}minus.png`; + toggledRows.forEach((el) => (el.style.display = "")); + } + }; + + const togglerElements = document.querySelectorAll("img.toggler"); + togglerElements.forEach((el) => + el.addEventListener("click", (event) => toggler(event.currentTarget)) + ); + togglerElements.forEach((el) => (el.style.display = "")); + if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) togglerElements.forEach(toggler); + }, + + initOnKeyListeners: () => { + // only install a listener if it is really needed + if ( + !DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS && + !DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS + ) + return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.altKey || event.ctrlKey || event.metaKey) return; + + if (!event.shiftKey) { + switch (event.key) { + case "ArrowLeft": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const prevLink = document.querySelector('link[rel="prev"]'); + if (prevLink && prevLink.href) { + window.location.href = prevLink.href; + event.preventDefault(); + } + break; + case "ArrowRight": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const nextLink = document.querySelector('link[rel="next"]'); + if (nextLink && nextLink.href) { + window.location.href = nextLink.href; + event.preventDefault(); + } + break; + } + } + + // some keyboard layouts may need Shift to get / + switch (event.key) { + case "/": + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) break; + Documentation.focusSearchBar(); + event.preventDefault(); + } + }); + }, +}; + +// quick alias for translations +const _ = Documentation.gettext; + +_ready(Documentation.init); diff --git a/_static/documentation_options.js b/_static/documentation_options.js new file mode 100644 index 000000000..da658f847 --- /dev/null +++ b/_static/documentation_options.js @@ -0,0 +1,13 @@ +const DOCUMENTATION_OPTIONS = { + VERSION: '2.3.0+26.g65e2074ad', + LANGUAGE: 'en', + COLLAPSE_INDEX: false, + BUILDER: 'html', + FILE_SUFFIX: '.html', + LINK_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt', + NAVIGATION_WITH_KEYS: false, + SHOW_SEARCH_SUMMARY: true, + ENABLE_SEARCH_SHORTCUTS: true, +}; \ No newline at end of file diff --git a/_static/file.png b/_static/file.png new file mode 100644 index 000000000..a858a410e Binary files /dev/null and b/_static/file.png differ diff --git a/_static/fonts/Lato/lato-bold.eot b/_static/fonts/Lato/lato-bold.eot new file mode 100644 index 000000000..3361183a4 Binary files /dev/null and b/_static/fonts/Lato/lato-bold.eot differ diff --git a/_static/fonts/Lato/lato-bold.ttf b/_static/fonts/Lato/lato-bold.ttf new file mode 100644 index 000000000..29f691d5e Binary files /dev/null and b/_static/fonts/Lato/lato-bold.ttf differ diff --git a/_static/fonts/Lato/lato-bold.woff b/_static/fonts/Lato/lato-bold.woff new file mode 100644 index 000000000..c6dff51f0 Binary files /dev/null and b/_static/fonts/Lato/lato-bold.woff differ diff --git a/_static/fonts/Lato/lato-bold.woff2 b/_static/fonts/Lato/lato-bold.woff2 new file mode 100644 index 000000000..bb195043c Binary files /dev/null and b/_static/fonts/Lato/lato-bold.woff2 differ diff --git a/_static/fonts/Lato/lato-bolditalic.eot b/_static/fonts/Lato/lato-bolditalic.eot new file mode 100644 index 000000000..3d4154936 Binary files /dev/null and b/_static/fonts/Lato/lato-bolditalic.eot differ diff --git a/_static/fonts/Lato/lato-bolditalic.ttf b/_static/fonts/Lato/lato-bolditalic.ttf new file mode 100644 index 000000000..f402040b3 Binary files /dev/null and b/_static/fonts/Lato/lato-bolditalic.ttf differ diff --git a/_static/fonts/Lato/lato-bolditalic.woff b/_static/fonts/Lato/lato-bolditalic.woff new file mode 100644 index 000000000..88ad05b9f Binary files /dev/null and b/_static/fonts/Lato/lato-bolditalic.woff differ diff --git a/_static/fonts/Lato/lato-bolditalic.woff2 b/_static/fonts/Lato/lato-bolditalic.woff2 new file mode 100644 index 000000000..c4e3d804b Binary files /dev/null and b/_static/fonts/Lato/lato-bolditalic.woff2 differ diff --git a/_static/fonts/Lato/lato-italic.eot b/_static/fonts/Lato/lato-italic.eot new file mode 100644 index 000000000..3f826421a Binary files /dev/null and b/_static/fonts/Lato/lato-italic.eot differ diff --git a/_static/fonts/Lato/lato-italic.ttf b/_static/fonts/Lato/lato-italic.ttf new file mode 100644 index 000000000..b4bfc9b24 Binary files /dev/null and b/_static/fonts/Lato/lato-italic.ttf differ diff --git a/_static/fonts/Lato/lato-italic.woff b/_static/fonts/Lato/lato-italic.woff new file mode 100644 index 000000000..76114bc03 Binary files /dev/null and b/_static/fonts/Lato/lato-italic.woff differ diff --git a/_static/fonts/Lato/lato-italic.woff2 b/_static/fonts/Lato/lato-italic.woff2 new file mode 100644 index 000000000..3404f37e2 Binary files /dev/null and b/_static/fonts/Lato/lato-italic.woff2 differ diff --git a/_static/fonts/Lato/lato-regular.eot b/_static/fonts/Lato/lato-regular.eot new file mode 100644 index 000000000..11e3f2a5f Binary files /dev/null and b/_static/fonts/Lato/lato-regular.eot differ diff --git a/_static/fonts/Lato/lato-regular.ttf b/_static/fonts/Lato/lato-regular.ttf new file mode 100644 index 000000000..74decd9eb Binary files /dev/null and b/_static/fonts/Lato/lato-regular.ttf differ diff --git a/_static/fonts/Lato/lato-regular.woff b/_static/fonts/Lato/lato-regular.woff new file mode 100644 index 000000000..ae1307ff5 Binary files /dev/null and b/_static/fonts/Lato/lato-regular.woff differ diff --git a/_static/fonts/Lato/lato-regular.woff2 b/_static/fonts/Lato/lato-regular.woff2 new file mode 100644 index 000000000..3bf984332 Binary files /dev/null and b/_static/fonts/Lato/lato-regular.woff2 differ diff --git a/_static/fonts/RobotoSlab/roboto-slab-v7-bold.eot b/_static/fonts/RobotoSlab/roboto-slab-v7-bold.eot new file mode 100644 index 000000000..79dc8efed Binary files /dev/null and b/_static/fonts/RobotoSlab/roboto-slab-v7-bold.eot differ diff --git a/_static/fonts/RobotoSlab/roboto-slab-v7-bold.ttf b/_static/fonts/RobotoSlab/roboto-slab-v7-bold.ttf new file mode 100644 index 000000000..df5d1df27 Binary files /dev/null and b/_static/fonts/RobotoSlab/roboto-slab-v7-bold.ttf differ diff --git a/_static/fonts/RobotoSlab/roboto-slab-v7-bold.woff b/_static/fonts/RobotoSlab/roboto-slab-v7-bold.woff new file mode 100644 index 000000000..6cb600001 Binary files /dev/null and b/_static/fonts/RobotoSlab/roboto-slab-v7-bold.woff differ diff --git a/_static/fonts/RobotoSlab/roboto-slab-v7-bold.woff2 b/_static/fonts/RobotoSlab/roboto-slab-v7-bold.woff2 new file mode 100644 index 000000000..7059e2314 Binary files /dev/null and b/_static/fonts/RobotoSlab/roboto-slab-v7-bold.woff2 differ diff --git a/_static/fonts/RobotoSlab/roboto-slab-v7-regular.eot b/_static/fonts/RobotoSlab/roboto-slab-v7-regular.eot new file mode 100644 index 000000000..2f7ca78a1 Binary files /dev/null and b/_static/fonts/RobotoSlab/roboto-slab-v7-regular.eot differ diff --git a/_static/fonts/RobotoSlab/roboto-slab-v7-regular.ttf b/_static/fonts/RobotoSlab/roboto-slab-v7-regular.ttf new file mode 100644 index 000000000..eb52a7907 Binary files /dev/null and b/_static/fonts/RobotoSlab/roboto-slab-v7-regular.ttf differ diff --git a/_static/fonts/RobotoSlab/roboto-slab-v7-regular.woff b/_static/fonts/RobotoSlab/roboto-slab-v7-regular.woff new file mode 100644 index 000000000..f815f63f9 Binary files /dev/null and b/_static/fonts/RobotoSlab/roboto-slab-v7-regular.woff differ diff --git a/_static/fonts/RobotoSlab/roboto-slab-v7-regular.woff2 b/_static/fonts/RobotoSlab/roboto-slab-v7-regular.woff2 new file mode 100644 index 000000000..f2c76e5bd Binary files /dev/null and b/_static/fonts/RobotoSlab/roboto-slab-v7-regular.woff2 differ diff --git a/_static/jquery.js b/_static/jquery.js new file mode 100644 index 000000000..c4c6022f2 --- /dev/null +++ b/_static/jquery.js @@ -0,0 +1,2 @@ +/*! jQuery v3.6.0 | (c) OpenJS Foundation and other contributors | jquery.org/license */ +!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(C,e){"use strict";var t=[],r=Object.getPrototypeOf,s=t.slice,g=t.flat?function(e){return t.flat.call(e)}:function(e){return t.concat.apply([],e)},u=t.push,i=t.indexOf,n={},o=n.toString,v=n.hasOwnProperty,a=v.toString,l=a.call(Object),y={},m=function(e){return"function"==typeof e&&"number"!=typeof e.nodeType&&"function"!=typeof e.item},x=function(e){return null!=e&&e===e.window},E=C.document,c={type:!0,src:!0,nonce:!0,noModule:!0};function b(e,t,n){var r,i,o=(n=n||E).createElement("script");if(o.text=e,t)for(r in c)(i=t[r]||t.getAttribute&&t.getAttribute(r))&&o.setAttribute(r,i);n.head.appendChild(o).parentNode.removeChild(o)}function w(e){return null==e?e+"":"object"==typeof e||"function"==typeof e?n[o.call(e)]||"object":typeof e}var f="3.6.0",S=function(e,t){return new S.fn.init(e,t)};function p(e){var t=!!e&&"length"in e&&e.length,n=w(e);return!m(e)&&!x(e)&&("array"===n||0===t||"number"==typeof t&&0+~]|"+M+")"+M+"*"),U=new RegExp(M+"|>"),X=new RegExp(F),V=new RegExp("^"+I+"$"),G={ID:new RegExp("^#("+I+")"),CLASS:new RegExp("^\\.("+I+")"),TAG:new RegExp("^("+I+"|[*])"),ATTR:new RegExp("^"+W),PSEUDO:new RegExp("^"+F),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+M+"*(even|odd|(([+-]|)(\\d*)n|)"+M+"*(?:([+-]|)"+M+"*(\\d+)|))"+M+"*\\)|)","i"),bool:new RegExp("^(?:"+R+")$","i"),needsContext:new RegExp("^"+M+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+M+"*((?:-\\d)?\\d*)"+M+"*\\)|)(?=[^-]|$)","i")},Y=/HTML$/i,Q=/^(?:input|select|textarea|button)$/i,J=/^h\d$/i,K=/^[^{]+\{\s*\[native \w/,Z=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,ee=/[+~]/,te=new RegExp("\\\\[\\da-fA-F]{1,6}"+M+"?|\\\\([^\\r\\n\\f])","g"),ne=function(e,t){var n="0x"+e.slice(1)-65536;return t||(n<0?String.fromCharCode(n+65536):String.fromCharCode(n>>10|55296,1023&n|56320))},re=/([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g,ie=function(e,t){return t?"\0"===e?"\ufffd":e.slice(0,-1)+"\\"+e.charCodeAt(e.length-1).toString(16)+" ":"\\"+e},oe=function(){T()},ae=be(function(e){return!0===e.disabled&&"fieldset"===e.nodeName.toLowerCase()},{dir:"parentNode",next:"legend"});try{H.apply(t=O.call(p.childNodes),p.childNodes),t[p.childNodes.length].nodeType}catch(e){H={apply:t.length?function(e,t){L.apply(e,O.call(t))}:function(e,t){var n=e.length,r=0;while(e[n++]=t[r++]);e.length=n-1}}}function se(t,e,n,r){var i,o,a,s,u,l,c,f=e&&e.ownerDocument,p=e?e.nodeType:9;if(n=n||[],"string"!=typeof t||!t||1!==p&&9!==p&&11!==p)return n;if(!r&&(T(e),e=e||C,E)){if(11!==p&&(u=Z.exec(t)))if(i=u[1]){if(9===p){if(!(a=e.getElementById(i)))return n;if(a.id===i)return n.push(a),n}else if(f&&(a=f.getElementById(i))&&y(e,a)&&a.id===i)return n.push(a),n}else{if(u[2])return H.apply(n,e.getElementsByTagName(t)),n;if((i=u[3])&&d.getElementsByClassName&&e.getElementsByClassName)return H.apply(n,e.getElementsByClassName(i)),n}if(d.qsa&&!N[t+" "]&&(!v||!v.test(t))&&(1!==p||"object"!==e.nodeName.toLowerCase())){if(c=t,f=e,1===p&&(U.test(t)||z.test(t))){(f=ee.test(t)&&ye(e.parentNode)||e)===e&&d.scope||((s=e.getAttribute("id"))?s=s.replace(re,ie):e.setAttribute("id",s=S)),o=(l=h(t)).length;while(o--)l[o]=(s?"#"+s:":scope")+" "+xe(l[o]);c=l.join(",")}try{return H.apply(n,f.querySelectorAll(c)),n}catch(e){N(t,!0)}finally{s===S&&e.removeAttribute("id")}}}return g(t.replace($,"$1"),e,n,r)}function ue(){var r=[];return function e(t,n){return r.push(t+" ")>b.cacheLength&&delete e[r.shift()],e[t+" "]=n}}function le(e){return e[S]=!0,e}function ce(e){var t=C.createElement("fieldset");try{return!!e(t)}catch(e){return!1}finally{t.parentNode&&t.parentNode.removeChild(t),t=null}}function fe(e,t){var n=e.split("|"),r=n.length;while(r--)b.attrHandle[n[r]]=t}function pe(e,t){var n=t&&e,r=n&&1===e.nodeType&&1===t.nodeType&&e.sourceIndex-t.sourceIndex;if(r)return r;if(n)while(n=n.nextSibling)if(n===t)return-1;return e?1:-1}function de(t){return function(e){return"input"===e.nodeName.toLowerCase()&&e.type===t}}function he(n){return function(e){var t=e.nodeName.toLowerCase();return("input"===t||"button"===t)&&e.type===n}}function ge(t){return function(e){return"form"in e?e.parentNode&&!1===e.disabled?"label"in e?"label"in e.parentNode?e.parentNode.disabled===t:e.disabled===t:e.isDisabled===t||e.isDisabled!==!t&&ae(e)===t:e.disabled===t:"label"in e&&e.disabled===t}}function ve(a){return le(function(o){return o=+o,le(function(e,t){var n,r=a([],e.length,o),i=r.length;while(i--)e[n=r[i]]&&(e[n]=!(t[n]=e[n]))})})}function ye(e){return e&&"undefined"!=typeof e.getElementsByTagName&&e}for(e in d=se.support={},i=se.isXML=function(e){var t=e&&e.namespaceURI,n=e&&(e.ownerDocument||e).documentElement;return!Y.test(t||n&&n.nodeName||"HTML")},T=se.setDocument=function(e){var t,n,r=e?e.ownerDocument||e:p;return r!=C&&9===r.nodeType&&r.documentElement&&(a=(C=r).documentElement,E=!i(C),p!=C&&(n=C.defaultView)&&n.top!==n&&(n.addEventListener?n.addEventListener("unload",oe,!1):n.attachEvent&&n.attachEvent("onunload",oe)),d.scope=ce(function(e){return a.appendChild(e).appendChild(C.createElement("div")),"undefined"!=typeof e.querySelectorAll&&!e.querySelectorAll(":scope fieldset div").length}),d.attributes=ce(function(e){return e.className="i",!e.getAttribute("className")}),d.getElementsByTagName=ce(function(e){return e.appendChild(C.createComment("")),!e.getElementsByTagName("*").length}),d.getElementsByClassName=K.test(C.getElementsByClassName),d.getById=ce(function(e){return a.appendChild(e).id=S,!C.getElementsByName||!C.getElementsByName(S).length}),d.getById?(b.filter.ID=function(e){var t=e.replace(te,ne);return function(e){return e.getAttribute("id")===t}},b.find.ID=function(e,t){if("undefined"!=typeof t.getElementById&&E){var n=t.getElementById(e);return n?[n]:[]}}):(b.filter.ID=function(e){var n=e.replace(te,ne);return function(e){var t="undefined"!=typeof e.getAttributeNode&&e.getAttributeNode("id");return t&&t.value===n}},b.find.ID=function(e,t){if("undefined"!=typeof t.getElementById&&E){var n,r,i,o=t.getElementById(e);if(o){if((n=o.getAttributeNode("id"))&&n.value===e)return[o];i=t.getElementsByName(e),r=0;while(o=i[r++])if((n=o.getAttributeNode("id"))&&n.value===e)return[o]}return[]}}),b.find.TAG=d.getElementsByTagName?function(e,t){return"undefined"!=typeof t.getElementsByTagName?t.getElementsByTagName(e):d.qsa?t.querySelectorAll(e):void 0}:function(e,t){var n,r=[],i=0,o=t.getElementsByTagName(e);if("*"===e){while(n=o[i++])1===n.nodeType&&r.push(n);return r}return o},b.find.CLASS=d.getElementsByClassName&&function(e,t){if("undefined"!=typeof t.getElementsByClassName&&E)return t.getElementsByClassName(e)},s=[],v=[],(d.qsa=K.test(C.querySelectorAll))&&(ce(function(e){var t;a.appendChild(e).innerHTML="",e.querySelectorAll("[msallowcapture^='']").length&&v.push("[*^$]="+M+"*(?:''|\"\")"),e.querySelectorAll("[selected]").length||v.push("\\["+M+"*(?:value|"+R+")"),e.querySelectorAll("[id~="+S+"-]").length||v.push("~="),(t=C.createElement("input")).setAttribute("name",""),e.appendChild(t),e.querySelectorAll("[name='']").length||v.push("\\["+M+"*name"+M+"*="+M+"*(?:''|\"\")"),e.querySelectorAll(":checked").length||v.push(":checked"),e.querySelectorAll("a#"+S+"+*").length||v.push(".#.+[+~]"),e.querySelectorAll("\\\f"),v.push("[\\r\\n\\f]")}),ce(function(e){e.innerHTML="";var t=C.createElement("input");t.setAttribute("type","hidden"),e.appendChild(t).setAttribute("name","D"),e.querySelectorAll("[name=d]").length&&v.push("name"+M+"*[*^$|!~]?="),2!==e.querySelectorAll(":enabled").length&&v.push(":enabled",":disabled"),a.appendChild(e).disabled=!0,2!==e.querySelectorAll(":disabled").length&&v.push(":enabled",":disabled"),e.querySelectorAll("*,:x"),v.push(",.*:")})),(d.matchesSelector=K.test(c=a.matches||a.webkitMatchesSelector||a.mozMatchesSelector||a.oMatchesSelector||a.msMatchesSelector))&&ce(function(e){d.disconnectedMatch=c.call(e,"*"),c.call(e,"[s!='']:x"),s.push("!=",F)}),v=v.length&&new RegExp(v.join("|")),s=s.length&&new RegExp(s.join("|")),t=K.test(a.compareDocumentPosition),y=t||K.test(a.contains)?function(e,t){var n=9===e.nodeType?e.documentElement:e,r=t&&t.parentNode;return e===r||!(!r||1!==r.nodeType||!(n.contains?n.contains(r):e.compareDocumentPosition&&16&e.compareDocumentPosition(r)))}:function(e,t){if(t)while(t=t.parentNode)if(t===e)return!0;return!1},j=t?function(e,t){if(e===t)return l=!0,0;var n=!e.compareDocumentPosition-!t.compareDocumentPosition;return n||(1&(n=(e.ownerDocument||e)==(t.ownerDocument||t)?e.compareDocumentPosition(t):1)||!d.sortDetached&&t.compareDocumentPosition(e)===n?e==C||e.ownerDocument==p&&y(p,e)?-1:t==C||t.ownerDocument==p&&y(p,t)?1:u?P(u,e)-P(u,t):0:4&n?-1:1)}:function(e,t){if(e===t)return l=!0,0;var n,r=0,i=e.parentNode,o=t.parentNode,a=[e],s=[t];if(!i||!o)return e==C?-1:t==C?1:i?-1:o?1:u?P(u,e)-P(u,t):0;if(i===o)return pe(e,t);n=e;while(n=n.parentNode)a.unshift(n);n=t;while(n=n.parentNode)s.unshift(n);while(a[r]===s[r])r++;return r?pe(a[r],s[r]):a[r]==p?-1:s[r]==p?1:0}),C},se.matches=function(e,t){return se(e,null,null,t)},se.matchesSelector=function(e,t){if(T(e),d.matchesSelector&&E&&!N[t+" "]&&(!s||!s.test(t))&&(!v||!v.test(t)))try{var n=c.call(e,t);if(n||d.disconnectedMatch||e.document&&11!==e.document.nodeType)return n}catch(e){N(t,!0)}return 0":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(e){return e[1]=e[1].replace(te,ne),e[3]=(e[3]||e[4]||e[5]||"").replace(te,ne),"~="===e[2]&&(e[3]=" "+e[3]+" "),e.slice(0,4)},CHILD:function(e){return e[1]=e[1].toLowerCase(),"nth"===e[1].slice(0,3)?(e[3]||se.error(e[0]),e[4]=+(e[4]?e[5]+(e[6]||1):2*("even"===e[3]||"odd"===e[3])),e[5]=+(e[7]+e[8]||"odd"===e[3])):e[3]&&se.error(e[0]),e},PSEUDO:function(e){var t,n=!e[6]&&e[2];return G.CHILD.test(e[0])?null:(e[3]?e[2]=e[4]||e[5]||"":n&&X.test(n)&&(t=h(n,!0))&&(t=n.indexOf(")",n.length-t)-n.length)&&(e[0]=e[0].slice(0,t),e[2]=n.slice(0,t)),e.slice(0,3))}},filter:{TAG:function(e){var t=e.replace(te,ne).toLowerCase();return"*"===e?function(){return!0}:function(e){return e.nodeName&&e.nodeName.toLowerCase()===t}},CLASS:function(e){var t=m[e+" "];return t||(t=new RegExp("(^|"+M+")"+e+"("+M+"|$)"))&&m(e,function(e){return t.test("string"==typeof e.className&&e.className||"undefined"!=typeof e.getAttribute&&e.getAttribute("class")||"")})},ATTR:function(n,r,i){return function(e){var t=se.attr(e,n);return null==t?"!="===r:!r||(t+="","="===r?t===i:"!="===r?t!==i:"^="===r?i&&0===t.indexOf(i):"*="===r?i&&-1:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i;function j(e,n,r){return m(n)?S.grep(e,function(e,t){return!!n.call(e,t,e)!==r}):n.nodeType?S.grep(e,function(e){return e===n!==r}):"string"!=typeof n?S.grep(e,function(e){return-1)[^>]*|#([\w-]+))$/;(S.fn.init=function(e,t,n){var r,i;if(!e)return this;if(n=n||D,"string"==typeof e){if(!(r="<"===e[0]&&">"===e[e.length-1]&&3<=e.length?[null,e,null]:q.exec(e))||!r[1]&&t)return!t||t.jquery?(t||n).find(e):this.constructor(t).find(e);if(r[1]){if(t=t instanceof S?t[0]:t,S.merge(this,S.parseHTML(r[1],t&&t.nodeType?t.ownerDocument||t:E,!0)),N.test(r[1])&&S.isPlainObject(t))for(r in t)m(this[r])?this[r](t[r]):this.attr(r,t[r]);return this}return(i=E.getElementById(r[2]))&&(this[0]=i,this.length=1),this}return e.nodeType?(this[0]=e,this.length=1,this):m(e)?void 0!==n.ready?n.ready(e):e(S):S.makeArray(e,this)}).prototype=S.fn,D=S(E);var L=/^(?:parents|prev(?:Until|All))/,H={children:!0,contents:!0,next:!0,prev:!0};function O(e,t){while((e=e[t])&&1!==e.nodeType);return e}S.fn.extend({has:function(e){var t=S(e,this),n=t.length;return this.filter(function(){for(var e=0;e\x20\t\r\n\f]*)/i,he=/^$|^module$|\/(?:java|ecma)script/i;ce=E.createDocumentFragment().appendChild(E.createElement("div")),(fe=E.createElement("input")).setAttribute("type","radio"),fe.setAttribute("checked","checked"),fe.setAttribute("name","t"),ce.appendChild(fe),y.checkClone=ce.cloneNode(!0).cloneNode(!0).lastChild.checked,ce.innerHTML="",y.noCloneChecked=!!ce.cloneNode(!0).lastChild.defaultValue,ce.innerHTML="",y.option=!!ce.lastChild;var ge={thead:[1,"","
"],col:[2,"","
"],tr:[2,"","
"],td:[3,"","
"],_default:[0,"",""]};function ve(e,t){var n;return n="undefined"!=typeof e.getElementsByTagName?e.getElementsByTagName(t||"*"):"undefined"!=typeof e.querySelectorAll?e.querySelectorAll(t||"*"):[],void 0===t||t&&A(e,t)?S.merge([e],n):n}function ye(e,t){for(var n=0,r=e.length;n",""]);var me=/<|&#?\w+;/;function xe(e,t,n,r,i){for(var o,a,s,u,l,c,f=t.createDocumentFragment(),p=[],d=0,h=e.length;d\s*$/g;function je(e,t){return A(e,"table")&&A(11!==t.nodeType?t:t.firstChild,"tr")&&S(e).children("tbody")[0]||e}function De(e){return e.type=(null!==e.getAttribute("type"))+"/"+e.type,e}function qe(e){return"true/"===(e.type||"").slice(0,5)?e.type=e.type.slice(5):e.removeAttribute("type"),e}function Le(e,t){var n,r,i,o,a,s;if(1===t.nodeType){if(Y.hasData(e)&&(s=Y.get(e).events))for(i in Y.remove(t,"handle events"),s)for(n=0,r=s[i].length;n").attr(n.scriptAttrs||{}).prop({charset:n.scriptCharset,src:n.url}).on("load error",i=function(e){r.remove(),i=null,e&&t("error"===e.type?404:200,e.type)}),E.head.appendChild(r[0])},abort:function(){i&&i()}}});var _t,zt=[],Ut=/(=)\?(?=&|$)|\?\?/;S.ajaxSetup({jsonp:"callback",jsonpCallback:function(){var e=zt.pop()||S.expando+"_"+wt.guid++;return this[e]=!0,e}}),S.ajaxPrefilter("json jsonp",function(e,t,n){var r,i,o,a=!1!==e.jsonp&&(Ut.test(e.url)?"url":"string"==typeof e.data&&0===(e.contentType||"").indexOf("application/x-www-form-urlencoded")&&Ut.test(e.data)&&"data");if(a||"jsonp"===e.dataTypes[0])return r=e.jsonpCallback=m(e.jsonpCallback)?e.jsonpCallback():e.jsonpCallback,a?e[a]=e[a].replace(Ut,"$1"+r):!1!==e.jsonp&&(e.url+=(Tt.test(e.url)?"&":"?")+e.jsonp+"="+r),e.converters["script json"]=function(){return o||S.error(r+" was not called"),o[0]},e.dataTypes[0]="json",i=C[r],C[r]=function(){o=arguments},n.always(function(){void 0===i?S(C).removeProp(r):C[r]=i,e[r]&&(e.jsonpCallback=t.jsonpCallback,zt.push(r)),o&&m(i)&&i(o[0]),o=i=void 0}),"script"}),y.createHTMLDocument=((_t=E.implementation.createHTMLDocument("").body).innerHTML="
",2===_t.childNodes.length),S.parseHTML=function(e,t,n){return"string"!=typeof e?[]:("boolean"==typeof t&&(n=t,t=!1),t||(y.createHTMLDocument?((r=(t=E.implementation.createHTMLDocument("")).createElement("base")).href=E.location.href,t.head.appendChild(r)):t=E),o=!n&&[],(i=N.exec(e))?[t.createElement(i[1])]:(i=xe([e],t,o),o&&o.length&&S(o).remove(),S.merge([],i.childNodes)));var r,i,o},S.fn.load=function(e,t,n){var r,i,o,a=this,s=e.indexOf(" ");return-1").append(S.parseHTML(e)).find(r):e)}).always(n&&function(e,t){a.each(function(){n.apply(this,o||[e.responseText,t,e])})}),this},S.expr.pseudos.animated=function(t){return S.grep(S.timers,function(e){return t===e.elem}).length},S.offset={setOffset:function(e,t,n){var r,i,o,a,s,u,l=S.css(e,"position"),c=S(e),f={};"static"===l&&(e.style.position="relative"),s=c.offset(),o=S.css(e,"top"),u=S.css(e,"left"),("absolute"===l||"fixed"===l)&&-1<(o+u).indexOf("auto")?(a=(r=c.position()).top,i=r.left):(a=parseFloat(o)||0,i=parseFloat(u)||0),m(t)&&(t=t.call(e,n,S.extend({},s))),null!=t.top&&(f.top=t.top-s.top+a),null!=t.left&&(f.left=t.left-s.left+i),"using"in t?t.using.call(e,f):c.css(f)}},S.fn.extend({offset:function(t){if(arguments.length)return void 0===t?this:this.each(function(e){S.offset.setOffset(this,t,e)});var e,n,r=this[0];return r?r.getClientRects().length?(e=r.getBoundingClientRect(),n=r.ownerDocument.defaultView,{top:e.top+n.pageYOffset,left:e.left+n.pageXOffset}):{top:0,left:0}:void 0},position:function(){if(this[0]){var e,t,n,r=this[0],i={top:0,left:0};if("fixed"===S.css(r,"position"))t=r.getBoundingClientRect();else{t=this.offset(),n=r.ownerDocument,e=r.offsetParent||n.documentElement;while(e&&(e===n.body||e===n.documentElement)&&"static"===S.css(e,"position"))e=e.parentNode;e&&e!==r&&1===e.nodeType&&((i=S(e).offset()).top+=S.css(e,"borderTopWidth",!0),i.left+=S.css(e,"borderLeftWidth",!0))}return{top:t.top-i.top-S.css(r,"marginTop",!0),left:t.left-i.left-S.css(r,"marginLeft",!0)}}},offsetParent:function(){return this.map(function(){var e=this.offsetParent;while(e&&"static"===S.css(e,"position"))e=e.offsetParent;return e||re})}}),S.each({scrollLeft:"pageXOffset",scrollTop:"pageYOffset"},function(t,i){var o="pageYOffset"===i;S.fn[t]=function(e){return $(this,function(e,t,n){var r;if(x(e)?r=e:9===e.nodeType&&(r=e.defaultView),void 0===n)return r?r[i]:e[t];r?r.scrollTo(o?r.pageXOffset:n,o?n:r.pageYOffset):e[t]=n},t,e,arguments.length)}}),S.each(["top","left"],function(e,n){S.cssHooks[n]=Fe(y.pixelPosition,function(e,t){if(t)return t=We(e,n),Pe.test(t)?S(e).position()[n]+"px":t})}),S.each({Height:"height",Width:"width"},function(a,s){S.each({padding:"inner"+a,content:s,"":"outer"+a},function(r,o){S.fn[o]=function(e,t){var n=arguments.length&&(r||"boolean"!=typeof e),i=r||(!0===e||!0===t?"margin":"border");return $(this,function(e,t,n){var r;return x(e)?0===o.indexOf("outer")?e["inner"+a]:e.document.documentElement["client"+a]:9===e.nodeType?(r=e.documentElement,Math.max(e.body["scroll"+a],r["scroll"+a],e.body["offset"+a],r["offset"+a],r["client"+a])):void 0===n?S.css(e,t,i):S.style(e,t,n,i)},s,n?e:void 0,n)}})}),S.each(["ajaxStart","ajaxStop","ajaxComplete","ajaxError","ajaxSuccess","ajaxSend"],function(e,t){S.fn[t]=function(e){return this.on(t,e)}}),S.fn.extend({bind:function(e,t,n){return this.on(e,null,t,n)},unbind:function(e,t){return this.off(e,null,t)},delegate:function(e,t,n,r){return this.on(t,e,n,r)},undelegate:function(e,t,n){return 1===arguments.length?this.off(e,"**"):this.off(t,e||"**",n)},hover:function(e,t){return this.mouseenter(e).mouseleave(t||e)}}),S.each("blur focus focusin focusout resize scroll click dblclick mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave change select submit keydown keypress keyup contextmenu".split(" "),function(e,n){S.fn[n]=function(e,t){return 0"),n("table.docutils.footnote").wrap("
"),n("table.docutils.citation").wrap("
"),n(".wy-menu-vertical ul").not(".simple").siblings("a").each((function(){var t=n(this);expand=n(''),expand.on("click",(function(n){return e.toggleCurrent(t),n.stopPropagation(),!1})),t.prepend(expand)}))},reset:function(){var n=encodeURI(window.location.hash)||"#";try{var e=$(".wy-menu-vertical"),t=e.find('[href="'+n+'"]');if(0===t.length){var i=$('.document [id="'+n.substring(1)+'"]').closest("div.section");0===(t=e.find('[href="#'+i.attr("id")+'"]')).length&&(t=e.find('[href="#"]'))}if(t.length>0){$(".wy-menu-vertical .current").removeClass("current").attr("aria-expanded","false"),t.addClass("current").attr("aria-expanded","true"),t.closest("li.toctree-l1").parent().addClass("current").attr("aria-expanded","true");for(let n=1;n<=10;n++)t.closest("li.toctree-l"+n).addClass("current").attr("aria-expanded","true");t[0].scrollIntoView()}}catch(n){console.log("Error expanding nav for anchor",n)}},onScroll:function(){this.winScroll=!1;var n=this.win.scrollTop(),e=n+this.winHeight,t=this.navBar.scrollTop()+(n-this.winPosition);n<0||e>this.docHeight||(this.navBar.scrollTop(t),this.winPosition=n)},onResize:function(){this.winResize=!1,this.winHeight=this.win.height(),this.docHeight=$(document).height()},hashChange:function(){this.linkScroll=!0,this.win.one("hashchange",(function(){this.linkScroll=!1}))},toggleCurrent:function(n){var e=n.closest("li");e.siblings("li.current").removeClass("current").attr("aria-expanded","false"),e.siblings().find("li.current").removeClass("current").attr("aria-expanded","false");var t=e.find("> ul li");t.length&&(t.removeClass("current").attr("aria-expanded","false"),e.toggleClass("current").attr("aria-expanded",(function(n,e){return"true"==e?"false":"true"})))}},"undefined"!=typeof window&&(window.SphinxRtdTheme={Navigation:n.exports.ThemeNav,StickyNav:n.exports.ThemeNav}),function(){for(var n=0,e=["ms","moz","webkit","o"],t=0;t +
Languages
+ ${config.projects.translations + .map( + (translation) => ` +
+ ${translation.language.code} +
+ `, + ) + .join("\n")} + + `; + return languagesHTML; + } + + function renderVersions(config) { + if (!config.versions.active.length) { + return ""; + } + const versionsHTML = ` +
+
Versions
+ ${config.versions.active + .map( + (version) => ` +
+ ${version.slug} +
+ `, + ) + .join("\n")} +
+ `; + return versionsHTML; + } + + function renderDownloads(config) { + if (!Object.keys(config.versions.current.downloads).length) { + return ""; + } + const downloadsNameDisplay = { + pdf: "PDF", + epub: "Epub", + htmlzip: "HTML", + }; + + const downloadsHTML = ` +
+
Downloads
+ ${Object.entries(config.versions.current.downloads) + .map( + ([name, url]) => ` +
+ ${downloadsNameDisplay[name]} +
+ `, + ) + .join("\n")} +
+ `; + return downloadsHTML; + } + + document.addEventListener("readthedocs-addons-data-ready", function (event) { + const config = event.detail.data(); + + const flyout = ` +
+ + Read the Docs + v: ${config.versions.current.slug} + + +
+
+ ${renderLanguages(config)} + ${renderVersions(config)} + ${renderDownloads(config)} +
+
On Read the Docs
+
+ Project Home +
+
+ Builds +
+
+ Downloads +
+
+
+
Search
+
+
+ +
+
+
+
+ + Hosted by Read the Docs + +
+
+ `; + + // Inject the generated flyout into the body HTML element. + document.body.insertAdjacentHTML("beforeend", flyout); + + // Trigger the Read the Docs Addons Search modal when clicking on the "Search docs" input from inside the flyout. + document + .querySelector("#flyout-search-form") + .addEventListener("focusin", () => { + const event = new CustomEvent("readthedocs-search-show"); + document.dispatchEvent(event); + }); + }) +} + +if (themeLanguageSelector || themeVersionSelector) { + function onSelectorSwitch(event) { + const option = event.target.selectedIndex; + const item = event.target.options[option]; + window.location.href = item.dataset.url; + } + + document.addEventListener("readthedocs-addons-data-ready", function (event) { + const config = event.detail.data(); + + const versionSwitch = document.querySelector( + "div.switch-menus > div.version-switch", + ); + if (themeVersionSelector) { + let versions = config.versions.active; + if (config.versions.current.hidden || config.versions.current.type === "external") { + versions.unshift(config.versions.current); + } + const versionSelect = ` + + `; + + versionSwitch.innerHTML = versionSelect; + versionSwitch.firstElementChild.addEventListener("change", onSelectorSwitch); + } + + const languageSwitch = document.querySelector( + "div.switch-menus > div.language-switch", + ); + + if (themeLanguageSelector) { + if (config.projects.translations.length) { + // Add the current language to the options on the selector + let languages = config.projects.translations.concat( + config.projects.current, + ); + languages = languages.sort((a, b) => + a.language.name.localeCompare(b.language.name), + ); + + const languageSelect = ` + + `; + + languageSwitch.innerHTML = languageSelect; + languageSwitch.firstElementChild.addEventListener("change", onSelectorSwitch); + } + else { + languageSwitch.remove(); + } + } + }); +} + +document.addEventListener("readthedocs-addons-data-ready", function (event) { + // Trigger the Read the Docs Addons Search modal when clicking on "Search docs" input from the topnav. + document + .querySelector("[role='search'] input") + .addEventListener("focusin", () => { + const event = new CustomEvent("readthedocs-search-show"); + document.dispatchEvent(event); + }); +}); \ No newline at end of file diff --git a/_static/language_data.js b/_static/language_data.js new file mode 100644 index 000000000..c7fe6c6fa --- /dev/null +++ b/_static/language_data.js @@ -0,0 +1,192 @@ +/* + * This script contains the language-specific data used by searchtools.js, + * namely the list of stopwords, stemmer, scorer and splitter. + */ + +var stopwords = ["a", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "near", "no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "there", "these", "they", "this", "to", "was", "will", "with"]; + + +/* Non-minified version is copied as a separate JS file, if available */ + +/** + * Porter Stemmer + */ +var Stemmer = function() { + + var step2list = { + ational: 'ate', + tional: 'tion', + enci: 'ence', + anci: 'ance', + izer: 'ize', + bli: 'ble', + alli: 'al', + entli: 'ent', + eli: 'e', + ousli: 'ous', + ization: 'ize', + ation: 'ate', + ator: 'ate', + alism: 'al', + iveness: 'ive', + fulness: 'ful', + ousness: 'ous', + aliti: 'al', + iviti: 'ive', + biliti: 'ble', + logi: 'log' + }; + + var step3list = { + icate: 'ic', + ative: '', + alize: 'al', + iciti: 'ic', + ical: 'ic', + ful: '', + ness: '' + }; + + var c = "[^aeiou]"; // consonant + var v = "[aeiouy]"; // vowel + var C = c + "[^aeiouy]*"; // consonant sequence + var V = v + "[aeiou]*"; // vowel sequence + + var mgr0 = "^(" + C + ")?" + V + C; // [C]VC... is m>0 + var meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$"; // [C]VC[V] is m=1 + var mgr1 = "^(" + C + ")?" + V + C + V + C; // [C]VCVC... is m>1 + var s_v = "^(" + C + ")?" + v; // vowel in stem + + this.stemWord = function (w) { + var stem; + var suffix; + var firstch; + var origword = w; + + if (w.length < 3) + return w; + + var re; + var re2; + var re3; + var re4; + + firstch = w.substr(0,1); + if (firstch == "y") + w = firstch.toUpperCase() + w.substr(1); + + // Step 1a + re = /^(.+?)(ss|i)es$/; + re2 = /^(.+?)([^s])s$/; + + if (re.test(w)) + w = w.replace(re,"$1$2"); + else if (re2.test(w)) + w = w.replace(re2,"$1$2"); + + // Step 1b + re = /^(.+?)eed$/; + re2 = /^(.+?)(ed|ing)$/; + if (re.test(w)) { + var fp = re.exec(w); + re = new RegExp(mgr0); + if (re.test(fp[1])) { + re = /.$/; + w = w.replace(re,""); + } + } + else if (re2.test(w)) { + var fp = re2.exec(w); + stem = fp[1]; + re2 = new RegExp(s_v); + if (re2.test(stem)) { + w = stem; + re2 = /(at|bl|iz)$/; + re3 = new RegExp("([^aeiouylsz])\\1$"); + re4 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re2.test(w)) + w = w + "e"; + else if (re3.test(w)) { + re = /.$/; + w = w.replace(re,""); + } + else if (re4.test(w)) + w = w + "e"; + } + } + + // Step 1c + re = /^(.+?)y$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + re = new RegExp(s_v); + if (re.test(stem)) + w = stem + "i"; + } + + // Step 2 + re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stem)) + w = stem + step2list[suffix]; + } + + // Step 3 + re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stem)) + w = stem + step3list[suffix]; + } + + // Step 4 + re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/; + re2 = /^(.+?)(s|t)(ion)$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + re = new RegExp(mgr1); + if (re.test(stem)) + w = stem; + } + else if (re2.test(w)) { + var fp = re2.exec(w); + stem = fp[1] + fp[2]; + re2 = new RegExp(mgr1); + if (re2.test(stem)) + w = stem; + } + + // Step 5 + re = /^(.+?)e$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + re = new RegExp(mgr1); + re2 = new RegExp(meq1); + re3 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re.test(stem) || (re2.test(stem) && !(re3.test(stem)))) + w = stem; + } + re = /ll$/; + re2 = new RegExp(mgr1); + if (re.test(w) && re2.test(w)) { + re = /.$/; + w = w.replace(re,""); + } + + // and turn initial Y back to y + if (firstch == "y") + w = firstch.toLowerCase() + w.substr(1); + return w; + } +} + diff --git a/_static/minus.png b/_static/minus.png new file mode 100644 index 000000000..d96755fda Binary files /dev/null and b/_static/minus.png differ diff --git a/_static/plus.png b/_static/plus.png new file mode 100644 index 000000000..7107cec93 Binary files /dev/null and b/_static/plus.png differ diff --git a/_static/pygments.css b/_static/pygments.css new file mode 100644 index 000000000..0d49244ed --- /dev/null +++ b/_static/pygments.css @@ -0,0 +1,75 @@ +pre { line-height: 125%; } +td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; } +span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; } +td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } +span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } +.highlight .hll { background-color: #ffffcc } +.highlight { background: #eeffcc; } +.highlight .c { color: #408090; font-style: italic } /* Comment */ +.highlight .err { border: 1px solid #FF0000 } /* Error */ +.highlight .k { color: #007020; font-weight: bold } /* Keyword */ +.highlight .o { color: #666666 } /* Operator */ +.highlight .ch { color: #408090; font-style: italic } /* Comment.Hashbang */ +.highlight .cm { color: #408090; font-style: italic } /* Comment.Multiline */ +.highlight .cp { color: #007020 } /* Comment.Preproc */ +.highlight .cpf { color: #408090; font-style: italic } /* Comment.PreprocFile */ +.highlight .c1 { color: #408090; font-style: italic } /* Comment.Single */ +.highlight .cs { color: #408090; background-color: #fff0f0 } /* Comment.Special */ +.highlight .gd { color: #A00000 } /* Generic.Deleted */ +.highlight .ge { font-style: italic } /* Generic.Emph */ +.highlight .ges { font-weight: bold; font-style: italic } /* Generic.EmphStrong */ +.highlight .gr { color: #FF0000 } /* Generic.Error */ +.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */ +.highlight .gi { color: #00A000 } /* Generic.Inserted */ +.highlight .go { color: #333333 } /* Generic.Output */ +.highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */ +.highlight .gs { font-weight: bold } /* Generic.Strong */ +.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ +.highlight .gt { color: #0044DD } /* Generic.Traceback */ +.highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */ +.highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */ +.highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */ +.highlight .kp { color: #007020 } /* Keyword.Pseudo */ +.highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */ +.highlight .kt { color: #902000 } /* Keyword.Type */ +.highlight .m { color: #208050 } /* Literal.Number */ +.highlight .s { color: #4070a0 } /* Literal.String */ +.highlight .na { color: #4070a0 } /* Name.Attribute */ +.highlight .nb { color: #007020 } /* Name.Builtin */ +.highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */ +.highlight .no { color: #60add5 } /* Name.Constant */ +.highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */ +.highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */ +.highlight .ne { color: #007020 } /* Name.Exception */ +.highlight .nf { color: #06287e } /* Name.Function */ +.highlight .nl { color: #002070; font-weight: bold } /* Name.Label */ +.highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */ +.highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */ +.highlight .nv { color: #bb60d5 } /* Name.Variable */ +.highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */ +.highlight .w { color: #bbbbbb } /* Text.Whitespace */ +.highlight .mb { color: #208050 } /* Literal.Number.Bin */ +.highlight .mf { color: #208050 } /* Literal.Number.Float */ +.highlight .mh { color: #208050 } /* Literal.Number.Hex */ +.highlight .mi { color: #208050 } /* Literal.Number.Integer */ +.highlight .mo { color: #208050 } /* Literal.Number.Oct */ +.highlight .sa { color: #4070a0 } /* Literal.String.Affix */ +.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */ +.highlight .sc { color: #4070a0 } /* Literal.String.Char */ +.highlight .dl { color: #4070a0 } /* Literal.String.Delimiter */ +.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */ +.highlight .s2 { color: #4070a0 } /* Literal.String.Double */ +.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */ +.highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */ +.highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */ +.highlight .sx { color: #c65d09 } /* Literal.String.Other */ +.highlight .sr { color: #235388 } /* Literal.String.Regex */ +.highlight .s1 { color: #4070a0 } /* Literal.String.Single */ +.highlight .ss { color: #517918 } /* Literal.String.Symbol */ +.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */ +.highlight .fm { color: #06287e } /* Name.Function.Magic */ +.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */ +.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */ +.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */ +.highlight .vm { color: #bb60d5 } /* Name.Variable.Magic */ +.highlight .il { color: #208050 } /* Literal.Number.Integer.Long */ \ No newline at end of file diff --git a/_static/release.html b/_static/release.html new file mode 100644 index 000000000..f261a08f4 --- /dev/null +++ b/_static/release.html @@ -0,0 +1,11 @@ + + + + + Latest release + + + +

+ + diff --git a/_static/release.svg b/_static/release.svg new file mode 100644 index 000000000..a2d187517 --- /dev/null +++ b/_static/release.svg @@ -0,0 +1,41 @@ + + + + + + + + + + + + + + + + + + + + + + release + + + release + + + 2.3.0 + + + 2.3.0 + + + + + diff --git a/_static/searchtools.js b/_static/searchtools.js new file mode 100644 index 000000000..2c774d17a --- /dev/null +++ b/_static/searchtools.js @@ -0,0 +1,632 @@ +/* + * Sphinx JavaScript utilities for the full-text search. + */ +"use strict"; + +/** + * Simple result scoring code. + */ +if (typeof Scorer === "undefined") { + var Scorer = { + // Implement the following function to further tweak the score for each result + // The function takes a result array [docname, title, anchor, descr, score, filename] + // and returns the new score. + /* + score: result => { + const [docname, title, anchor, descr, score, filename, kind] = result + return score + }, + */ + + // query matches the full name of an object + objNameMatch: 11, + // or matches in the last dotted part of the object name + objPartialMatch: 6, + // Additive scores depending on the priority of the object + objPrio: { + 0: 15, // used to be importantResults + 1: 5, // used to be objectResults + 2: -5, // used to be unimportantResults + }, + // Used when the priority is not in the mapping. + objPrioDefault: 0, + + // query found in title + title: 15, + partialTitle: 7, + // query found in terms + term: 5, + partialTerm: 2, + }; +} + +// Global search result kind enum, used by themes to style search results. +class SearchResultKind { + static get index() { return "index"; } + static get object() { return "object"; } + static get text() { return "text"; } + static get title() { return "title"; } +} + +const _removeChildren = (element) => { + while (element && element.lastChild) element.removeChild(element.lastChild); +}; + +/** + * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions#escaping + */ +const _escapeRegExp = (string) => + string.replace(/[.*+\-?^${}()|[\]\\]/g, "\\$&"); // $& means the whole matched string + +const _displayItem = (item, searchTerms, highlightTerms) => { + const docBuilder = DOCUMENTATION_OPTIONS.BUILDER; + const docFileSuffix = DOCUMENTATION_OPTIONS.FILE_SUFFIX; + const docLinkSuffix = DOCUMENTATION_OPTIONS.LINK_SUFFIX; + const showSearchSummary = DOCUMENTATION_OPTIONS.SHOW_SEARCH_SUMMARY; + const contentRoot = document.documentElement.dataset.content_root; + + const [docName, title, anchor, descr, score, _filename, kind] = item; + + let listItem = document.createElement("li"); + // Add a class representing the item's type: + // can be used by a theme's CSS selector for styling + // See SearchResultKind for the class names. + listItem.classList.add(`kind-${kind}`); + let requestUrl; + let linkUrl; + if (docBuilder === "dirhtml") { + // dirhtml builder + let dirname = docName + "/"; + if (dirname.match(/\/index\/$/)) + dirname = dirname.substring(0, dirname.length - 6); + else if (dirname === "index/") dirname = ""; + requestUrl = contentRoot + dirname; + linkUrl = requestUrl; + } else { + // normal html builders + requestUrl = contentRoot + docName + docFileSuffix; + linkUrl = docName + docLinkSuffix; + } + let linkEl = listItem.appendChild(document.createElement("a")); + linkEl.href = linkUrl + anchor; + linkEl.dataset.score = score; + linkEl.innerHTML = title; + if (descr) { + listItem.appendChild(document.createElement("span")).innerHTML = + " (" + descr + ")"; + // highlight search terms in the description + if (SPHINX_HIGHLIGHT_ENABLED) // set in sphinx_highlight.js + highlightTerms.forEach((term) => _highlightText(listItem, term, "highlighted")); + } + else if (showSearchSummary) + fetch(requestUrl) + .then((responseData) => responseData.text()) + .then((data) => { + if (data) + listItem.appendChild( + Search.makeSearchSummary(data, searchTerms, anchor) + ); + // highlight search terms in the summary + if (SPHINX_HIGHLIGHT_ENABLED) // set in sphinx_highlight.js + highlightTerms.forEach((term) => _highlightText(listItem, term, "highlighted")); + }); + Search.output.appendChild(listItem); +}; +const _finishSearch = (resultCount) => { + Search.stopPulse(); + Search.title.innerText = _("Search Results"); + if (!resultCount) + Search.status.innerText = Documentation.gettext( + "Your search did not match any documents. Please make sure that all words are spelled correctly and that you've selected enough categories." + ); + else + Search.status.innerText = Documentation.ngettext( + "Search finished, found one page matching the search query.", + "Search finished, found ${resultCount} pages matching the search query.", + resultCount, + ).replace('${resultCount}', resultCount); +}; +const _displayNextItem = ( + results, + resultCount, + searchTerms, + highlightTerms, +) => { + // results left, load the summary and display it + // this is intended to be dynamic (don't sub resultsCount) + if (results.length) { + _displayItem(results.pop(), searchTerms, highlightTerms); + setTimeout( + () => _displayNextItem(results, resultCount, searchTerms, highlightTerms), + 5 + ); + } + // search finished, update title and status message + else _finishSearch(resultCount); +}; +// Helper function used by query() to order search results. +// Each input is an array of [docname, title, anchor, descr, score, filename, kind]. +// Order the results by score (in opposite order of appearance, since the +// `_displayNextItem` function uses pop() to retrieve items) and then alphabetically. +const _orderResultsByScoreThenName = (a, b) => { + const leftScore = a[4]; + const rightScore = b[4]; + if (leftScore === rightScore) { + // same score: sort alphabetically + const leftTitle = a[1].toLowerCase(); + const rightTitle = b[1].toLowerCase(); + if (leftTitle === rightTitle) return 0; + return leftTitle > rightTitle ? -1 : 1; // inverted is intentional + } + return leftScore > rightScore ? 1 : -1; +}; + +/** + * Default splitQuery function. Can be overridden in ``sphinx.search`` with a + * custom function per language. + * + * The regular expression works by splitting the string on consecutive characters + * that are not Unicode letters, numbers, underscores, or emoji characters. + * This is the same as ``\W+`` in Python, preserving the surrogate pair area. + */ +if (typeof splitQuery === "undefined") { + var splitQuery = (query) => query + .split(/[^\p{Letter}\p{Number}_\p{Emoji_Presentation}]+/gu) + .filter(term => term) // remove remaining empty strings +} + +/** + * Search Module + */ +const Search = { + _index: null, + _queued_query: null, + _pulse_status: -1, + + htmlToText: (htmlString, anchor) => { + const htmlElement = new DOMParser().parseFromString(htmlString, 'text/html'); + for (const removalQuery of [".headerlink", "script", "style"]) { + htmlElement.querySelectorAll(removalQuery).forEach((el) => { el.remove() }); + } + if (anchor) { + const anchorContent = htmlElement.querySelector(`[role="main"] ${anchor}`); + if (anchorContent) return anchorContent.textContent; + + console.warn( + `Anchored content block not found. Sphinx search tries to obtain it via DOM query '[role=main] ${anchor}'. Check your theme or template.` + ); + } + + // if anchor not specified or not found, fall back to main content + const docContent = htmlElement.querySelector('[role="main"]'); + if (docContent) return docContent.textContent; + + console.warn( + "Content block not found. Sphinx search tries to obtain it via DOM query '[role=main]'. Check your theme or template." + ); + return ""; + }, + + init: () => { + const query = new URLSearchParams(window.location.search).get("q"); + document + .querySelectorAll('input[name="q"]') + .forEach((el) => (el.value = query)); + if (query) Search.performSearch(query); + }, + + loadIndex: (url) => + (document.body.appendChild(document.createElement("script")).src = url), + + setIndex: (index) => { + Search._index = index; + if (Search._queued_query !== null) { + const query = Search._queued_query; + Search._queued_query = null; + Search.query(query); + } + }, + + hasIndex: () => Search._index !== null, + + deferQuery: (query) => (Search._queued_query = query), + + stopPulse: () => (Search._pulse_status = -1), + + startPulse: () => { + if (Search._pulse_status >= 0) return; + + const pulse = () => { + Search._pulse_status = (Search._pulse_status + 1) % 4; + Search.dots.innerText = ".".repeat(Search._pulse_status); + if (Search._pulse_status >= 0) window.setTimeout(pulse, 500); + }; + pulse(); + }, + + /** + * perform a search for something (or wait until index is loaded) + */ + performSearch: (query) => { + // create the required interface elements + const searchText = document.createElement("h2"); + searchText.textContent = _("Searching"); + const searchSummary = document.createElement("p"); + searchSummary.classList.add("search-summary"); + searchSummary.innerText = ""; + const searchList = document.createElement("ul"); + searchList.setAttribute("role", "list"); + searchList.classList.add("search"); + + const out = document.getElementById("search-results"); + Search.title = out.appendChild(searchText); + Search.dots = Search.title.appendChild(document.createElement("span")); + Search.status = out.appendChild(searchSummary); + Search.output = out.appendChild(searchList); + + const searchProgress = document.getElementById("search-progress"); + // Some themes don't use the search progress node + if (searchProgress) { + searchProgress.innerText = _("Preparing search..."); + } + Search.startPulse(); + + // index already loaded, the browser was quick! + if (Search.hasIndex()) Search.query(query); + else Search.deferQuery(query); + }, + + _parseQuery: (query) => { + // stem the search terms and add them to the correct list + const stemmer = new Stemmer(); + const searchTerms = new Set(); + const excludedTerms = new Set(); + const highlightTerms = new Set(); + const objectTerms = new Set(splitQuery(query.toLowerCase().trim())); + splitQuery(query.trim()).forEach((queryTerm) => { + const queryTermLower = queryTerm.toLowerCase(); + + // maybe skip this "word" + // stopwords array is from language_data.js + if ( + stopwords.indexOf(queryTermLower) !== -1 || + queryTerm.match(/^\d+$/) + ) + return; + + // stem the word + let word = stemmer.stemWord(queryTermLower); + // select the correct list + if (word[0] === "-") excludedTerms.add(word.substr(1)); + else { + searchTerms.add(word); + highlightTerms.add(queryTermLower); + } + }); + + if (SPHINX_HIGHLIGHT_ENABLED) { // set in sphinx_highlight.js + localStorage.setItem("sphinx_highlight_terms", [...highlightTerms].join(" ")) + } + + // console.debug("SEARCH: searching for:"); + // console.info("required: ", [...searchTerms]); + // console.info("excluded: ", [...excludedTerms]); + + return [query, searchTerms, excludedTerms, highlightTerms, objectTerms]; + }, + + /** + * execute search (requires search index to be loaded) + */ + _performSearch: (query, searchTerms, excludedTerms, highlightTerms, objectTerms) => { + const filenames = Search._index.filenames; + const docNames = Search._index.docnames; + const titles = Search._index.titles; + const allTitles = Search._index.alltitles; + const indexEntries = Search._index.indexentries; + + // Collect multiple result groups to be sorted separately and then ordered. + // Each is an array of [docname, title, anchor, descr, score, filename, kind]. + const normalResults = []; + const nonMainIndexResults = []; + + _removeChildren(document.getElementById("search-progress")); + + const queryLower = query.toLowerCase().trim(); + for (const [title, foundTitles] of Object.entries(allTitles)) { + if (title.toLowerCase().trim().includes(queryLower) && (queryLower.length >= title.length/2)) { + for (const [file, id] of foundTitles) { + const score = Math.round(Scorer.title * queryLower.length / title.length); + const boost = titles[file] === title ? 1 : 0; // add a boost for document titles + normalResults.push([ + docNames[file], + titles[file] !== title ? `${titles[file]} > ${title}` : title, + id !== null ? "#" + id : "", + null, + score + boost, + filenames[file], + SearchResultKind.title, + ]); + } + } + } + + // search for explicit entries in index directives + for (const [entry, foundEntries] of Object.entries(indexEntries)) { + if (entry.includes(queryLower) && (queryLower.length >= entry.length/2)) { + for (const [file, id, isMain] of foundEntries) { + const score = Math.round(100 * queryLower.length / entry.length); + const result = [ + docNames[file], + titles[file], + id ? "#" + id : "", + null, + score, + filenames[file], + SearchResultKind.index, + ]; + if (isMain) { + normalResults.push(result); + } else { + nonMainIndexResults.push(result); + } + } + } + } + + // lookup as object + objectTerms.forEach((term) => + normalResults.push(...Search.performObjectSearch(term, objectTerms)) + ); + + // lookup as search terms in fulltext + normalResults.push(...Search.performTermsSearch(searchTerms, excludedTerms)); + + // let the scorer override scores with a custom scoring function + if (Scorer.score) { + normalResults.forEach((item) => (item[4] = Scorer.score(item))); + nonMainIndexResults.forEach((item) => (item[4] = Scorer.score(item))); + } + + // Sort each group of results by score and then alphabetically by name. + normalResults.sort(_orderResultsByScoreThenName); + nonMainIndexResults.sort(_orderResultsByScoreThenName); + + // Combine the result groups in (reverse) order. + // Non-main index entries are typically arbitrary cross-references, + // so display them after other results. + let results = [...nonMainIndexResults, ...normalResults]; + + // remove duplicate search results + // note the reversing of results, so that in the case of duplicates, the highest-scoring entry is kept + let seen = new Set(); + results = results.reverse().reduce((acc, result) => { + let resultStr = result.slice(0, 4).concat([result[5]]).map(v => String(v)).join(','); + if (!seen.has(resultStr)) { + acc.push(result); + seen.add(resultStr); + } + return acc; + }, []); + + return results.reverse(); + }, + + query: (query) => { + const [searchQuery, searchTerms, excludedTerms, highlightTerms, objectTerms] = Search._parseQuery(query); + const results = Search._performSearch(searchQuery, searchTerms, excludedTerms, highlightTerms, objectTerms); + + // for debugging + //Search.lastresults = results.slice(); // a copy + // console.info("search results:", Search.lastresults); + + // print the results + _displayNextItem(results, results.length, searchTerms, highlightTerms); + }, + + /** + * search for object names + */ + performObjectSearch: (object, objectTerms) => { + const filenames = Search._index.filenames; + const docNames = Search._index.docnames; + const objects = Search._index.objects; + const objNames = Search._index.objnames; + const titles = Search._index.titles; + + const results = []; + + const objectSearchCallback = (prefix, match) => { + const name = match[4] + const fullname = (prefix ? prefix + "." : "") + name; + const fullnameLower = fullname.toLowerCase(); + if (fullnameLower.indexOf(object) < 0) return; + + let score = 0; + const parts = fullnameLower.split("."); + + // check for different match types: exact matches of full name or + // "last name" (i.e. last dotted part) + if (fullnameLower === object || parts.slice(-1)[0] === object) + score += Scorer.objNameMatch; + else if (parts.slice(-1)[0].indexOf(object) > -1) + score += Scorer.objPartialMatch; // matches in last name + + const objName = objNames[match[1]][2]; + const title = titles[match[0]]; + + // If more than one term searched for, we require other words to be + // found in the name/title/description + const otherTerms = new Set(objectTerms); + otherTerms.delete(object); + if (otherTerms.size > 0) { + const haystack = `${prefix} ${name} ${objName} ${title}`.toLowerCase(); + if ( + [...otherTerms].some((otherTerm) => haystack.indexOf(otherTerm) < 0) + ) + return; + } + + let anchor = match[3]; + if (anchor === "") anchor = fullname; + else if (anchor === "-") anchor = objNames[match[1]][1] + "-" + fullname; + + const descr = objName + _(", in ") + title; + + // add custom score for some objects according to scorer + if (Scorer.objPrio.hasOwnProperty(match[2])) + score += Scorer.objPrio[match[2]]; + else score += Scorer.objPrioDefault; + + results.push([ + docNames[match[0]], + fullname, + "#" + anchor, + descr, + score, + filenames[match[0]], + SearchResultKind.object, + ]); + }; + Object.keys(objects).forEach((prefix) => + objects[prefix].forEach((array) => + objectSearchCallback(prefix, array) + ) + ); + return results; + }, + + /** + * search for full-text terms in the index + */ + performTermsSearch: (searchTerms, excludedTerms) => { + // prepare search + const terms = Search._index.terms; + const titleTerms = Search._index.titleterms; + const filenames = Search._index.filenames; + const docNames = Search._index.docnames; + const titles = Search._index.titles; + + const scoreMap = new Map(); + const fileMap = new Map(); + + // perform the search on the required terms + searchTerms.forEach((word) => { + const files = []; + const arr = [ + { files: terms[word], score: Scorer.term }, + { files: titleTerms[word], score: Scorer.title }, + ]; + // add support for partial matches + if (word.length > 2) { + const escapedWord = _escapeRegExp(word); + if (!terms.hasOwnProperty(word)) { + Object.keys(terms).forEach((term) => { + if (term.match(escapedWord)) + arr.push({ files: terms[term], score: Scorer.partialTerm }); + }); + } + if (!titleTerms.hasOwnProperty(word)) { + Object.keys(titleTerms).forEach((term) => { + if (term.match(escapedWord)) + arr.push({ files: titleTerms[term], score: Scorer.partialTitle }); + }); + } + } + + // no match but word was a required one + if (arr.every((record) => record.files === undefined)) return; + + // found search word in contents + arr.forEach((record) => { + if (record.files === undefined) return; + + let recordFiles = record.files; + if (recordFiles.length === undefined) recordFiles = [recordFiles]; + files.push(...recordFiles); + + // set score for the word in each file + recordFiles.forEach((file) => { + if (!scoreMap.has(file)) scoreMap.set(file, {}); + scoreMap.get(file)[word] = record.score; + }); + }); + + // create the mapping + files.forEach((file) => { + if (!fileMap.has(file)) fileMap.set(file, [word]); + else if (fileMap.get(file).indexOf(word) === -1) fileMap.get(file).push(word); + }); + }); + + // now check if the files don't contain excluded terms + const results = []; + for (const [file, wordList] of fileMap) { + // check if all requirements are matched + + // as search terms with length < 3 are discarded + const filteredTermCount = [...searchTerms].filter( + (term) => term.length > 2 + ).length; + if ( + wordList.length !== searchTerms.size && + wordList.length !== filteredTermCount + ) + continue; + + // ensure that none of the excluded terms is in the search result + if ( + [...excludedTerms].some( + (term) => + terms[term] === file || + titleTerms[term] === file || + (terms[term] || []).includes(file) || + (titleTerms[term] || []).includes(file) + ) + ) + break; + + // select one (max) score for the file. + const score = Math.max(...wordList.map((w) => scoreMap.get(file)[w])); + // add result to the result list + results.push([ + docNames[file], + titles[file], + "", + null, + score, + filenames[file], + SearchResultKind.text, + ]); + } + return results; + }, + + /** + * helper function to return a node containing the + * search summary for a given text. keywords is a list + * of stemmed words. + */ + makeSearchSummary: (htmlText, keywords, anchor) => { + const text = Search.htmlToText(htmlText, anchor); + if (text === "") return null; + + const textLower = text.toLowerCase(); + const actualStartPosition = [...keywords] + .map((k) => textLower.indexOf(k.toLowerCase())) + .filter((i) => i > -1) + .slice(-1)[0]; + const startWithContext = Math.max(actualStartPosition - 120, 0); + + const top = startWithContext === 0 ? "" : "..."; + const tail = startWithContext + 240 < text.length ? "..." : ""; + + let summary = document.createElement("p"); + summary.classList.add("context"); + summary.textContent = top + text.substr(startWithContext, 240).trim() + tail; + + return summary; + }, +}; + +_ready(Search.init); diff --git a/_static/sphinx_highlight.js b/_static/sphinx_highlight.js new file mode 100644 index 000000000..8a96c69a1 --- /dev/null +++ b/_static/sphinx_highlight.js @@ -0,0 +1,154 @@ +/* Highlighting utilities for Sphinx HTML documentation. */ +"use strict"; + +const SPHINX_HIGHLIGHT_ENABLED = true + +/** + * highlight a given string on a node by wrapping it in + * span elements with the given class name. + */ +const _highlight = (node, addItems, text, className) => { + if (node.nodeType === Node.TEXT_NODE) { + const val = node.nodeValue; + const parent = node.parentNode; + const pos = val.toLowerCase().indexOf(text); + if ( + pos >= 0 && + !parent.classList.contains(className) && + !parent.classList.contains("nohighlight") + ) { + let span; + + const closestNode = parent.closest("body, svg, foreignObject"); + const isInSVG = closestNode && closestNode.matches("svg"); + if (isInSVG) { + span = document.createElementNS("http://www.w3.org/2000/svg", "tspan"); + } else { + span = document.createElement("span"); + span.classList.add(className); + } + + span.appendChild(document.createTextNode(val.substr(pos, text.length))); + const rest = document.createTextNode(val.substr(pos + text.length)); + parent.insertBefore( + span, + parent.insertBefore( + rest, + node.nextSibling + ) + ); + node.nodeValue = val.substr(0, pos); + /* There may be more occurrences of search term in this node. So call this + * function recursively on the remaining fragment. + */ + _highlight(rest, addItems, text, className); + + if (isInSVG) { + const rect = document.createElementNS( + "http://www.w3.org/2000/svg", + "rect" + ); + const bbox = parent.getBBox(); + rect.x.baseVal.value = bbox.x; + rect.y.baseVal.value = bbox.y; + rect.width.baseVal.value = bbox.width; + rect.height.baseVal.value = bbox.height; + rect.setAttribute("class", className); + addItems.push({ parent: parent, target: rect }); + } + } + } else if (node.matches && !node.matches("button, select, textarea")) { + node.childNodes.forEach((el) => _highlight(el, addItems, text, className)); + } +}; +const _highlightText = (thisNode, text, className) => { + let addItems = []; + _highlight(thisNode, addItems, text, className); + addItems.forEach((obj) => + obj.parent.insertAdjacentElement("beforebegin", obj.target) + ); +}; + +/** + * Small JavaScript module for the documentation. + */ +const SphinxHighlight = { + + /** + * highlight the search words provided in localstorage in the text + */ + highlightSearchWords: () => { + if (!SPHINX_HIGHLIGHT_ENABLED) return; // bail if no highlight + + // get and clear terms from localstorage + const url = new URL(window.location); + const highlight = + localStorage.getItem("sphinx_highlight_terms") + || url.searchParams.get("highlight") + || ""; + localStorage.removeItem("sphinx_highlight_terms") + url.searchParams.delete("highlight"); + window.history.replaceState({}, "", url); + + // get individual terms from highlight string + const terms = highlight.toLowerCase().split(/\s+/).filter(x => x); + if (terms.length === 0) return; // nothing to do + + // There should never be more than one element matching "div.body" + const divBody = document.querySelectorAll("div.body"); + const body = divBody.length ? divBody[0] : document.querySelector("body"); + window.setTimeout(() => { + terms.forEach((term) => _highlightText(body, term, "highlighted")); + }, 10); + + const searchBox = document.getElementById("searchbox"); + if (searchBox === null) return; + searchBox.appendChild( + document + .createRange() + .createContextualFragment( + '

' + + '' + + _("Hide Search Matches") + + "

" + ) + ); + }, + + /** + * helper function to hide the search marks again + */ + hideSearchWords: () => { + document + .querySelectorAll("#searchbox .highlight-link") + .forEach((el) => el.remove()); + document + .querySelectorAll("span.highlighted") + .forEach((el) => el.classList.remove("highlighted")); + localStorage.removeItem("sphinx_highlight_terms") + }, + + initEscapeListener: () => { + // only install a listener if it is really needed + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.shiftKey || event.altKey || event.ctrlKey || event.metaKey) return; + if (DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS && (event.key === "Escape")) { + SphinxHighlight.hideSearchWords(); + event.preventDefault(); + } + }); + }, +}; + +_ready(() => { + /* Do not call highlightSearchWords() when we are on the search page. + * It will highlight words from the *previous* search query. + */ + if (typeof Search === "undefined") SphinxHighlight.highlightSearchWords(); + SphinxHighlight.initEscapeListener(); +}); diff --git a/additional_docker.html b/additional_docker.html new file mode 100644 index 000000000..8c168cb13 --- /dev/null +++ b/additional_docker.html @@ -0,0 +1,167 @@ + + + + + + + + + BuildStream and Docker — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

BuildStream and Docker

+

BuildStream integrates with Docker in multiple ways. Here are some ways in +which these integrations work.

+
+

Run BuildStream inside Docker

+

Refer to the BuildStream inside Docker +documentation for instructions on how to run BuildStream as a Docker container.

+
+
+

Generate Docker images

+

The bst-docker-import script +can be used to generate a Docker image from built artifacts.

+

You can download it and make it executable like this:

+
mkdir -p ~/.local/bin
+curl --get https://raw.githubusercontent.com/apache/buildstream/master/contrib/bst-docker-import > ~/.local/bin/bst-docker-import
+chmod +x ~/.local/bin/bst-docker-import
+
+
+

Check if ~/.local/bin appears in your PATH environment variable – if it +doesn’t, you should +edit your ~/.profile so that it does.

+

Once the script is available in your PATH and assuming you have Docker +installed, you can start using the bst-docker-import script. Here is a +minimal example to generate an image called bst-hello from an element +called hello.bst assuming it is already built:

+
bst-docker-import -t bst-hello hello.bst
+
+
+

This script can also be used if you are running BuildStream inside Docker. In +this case, you will need to supply the command that you are using to run +BuildStream using the -c option. If you are using the +bst-here wrapper script, +you can achieve the same results as the above example like this:

+
bst-docker-import -c bst-here -t bst-hello hello.bst
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_cachekeys.html b/arch_cachekeys.html new file mode 100644 index 000000000..57acd3a4f --- /dev/null +++ b/arch_cachekeys.html @@ -0,0 +1,216 @@ + + + + + + + + + Cache keys — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Cache keys

+

Cache keys for artifacts are generated from the inputs of the build process +for the purpose of reusing artifacts in a well-defined, predictable way.

+
+

Structure

+

Cache keys are SHA256 hash values generated from a UTF-8 JSON document that +includes:

+
    +

  • Environment (e.g., project configuration and variables).

    • +

  • Element configuration (details depend on element kind, Element.get_unique_key()).

    • +

  • Sources (Source.get_unique_key()).

    • +

  • Dependencies (depending on cache key type, see below).

    • +

  • Public data.

    • +
+
+
+

Cache key types

+

There are two types of cache keys in BuildStream, strong and weak.

+

The purpose of a strong cache key is to capture the state of as many aspects +as possible that can have an influence on the build output. The aim is that +builds will be fully reproducible as long as the cache key doesn’t change, +with suitable module build systems that don’t embed timestamps, for example.

+

A strong cache key includes the strong cache key of each build dependency +(and their runtime dependencies) of the element as changes in build dependencies +(or their runtime dependencies) can result in build differences in reverse +dependencies. This means that whenever the strong cache key of a dependency +changes, the strong cache key of its reverse dependencies will change as well.

+

A weak cache key has an almost identical structure, however, it includes +only the names of build dependencies, not their cache keys or their runtime +dependencies. A weak cache key will thus still change when the element itself +or the environment changes but it will not change when a dependency is updated.

+

For elements without build dependencies the strong cache key is identical +to the weak cache key.

+

Note that dependencies which are not required at build time do not affect +either kind of key.

+
+
+

Strict build plan

+

This is the default build plan that exclusively uses strong cache keys +for the core functionality. An element’s cache key can be calculated when +the cache keys of the element’s build dependencies (and their runtime +dependencies) have been calculated and either tracking is not enabled or it +has already completed for this element, i.e., the ref is available. +This means that with tracking disabled the cache keys of all elements could be +calculated right at the start of a build session.

+

While BuildStream only uses strong cache keys with the strict build plan +for the actual staging and build process, it will still calculate weak +cache keys for each element. This allows BuildStream to store the artifact +in the cache with both keys, reducing rebuilds when switching between strict +and non-strict build plans. If the artifact cache already contains an +artifact with the same weak cache key, it’s replaced. Thus, non-strict +builds always use the latest artifact available for a given weak cache key.

+
+
+

Non-strict build plan

+

The non-strict build plan disables the time-consuming automatic rebuild of +reverse dependencies at the cost of dropping the reproducibility benefits. +It uses the weak cache keys for the core staging and build process. +I.e., if an artifact is available with the calculated weak cache key, +it will be reused for staging instead of being rebuilt. weak cache keys +can be calculated early in the build session. After tracking, similar to +when strong cache keys can be calculated with a strict build plan.

+

Similar to how strict build plans also calculate weak cache keys, non-strict +build plans also calculate strong cache keys. However, this is slightly +more complex. To calculate the strong cache key of an element, BuildStream +requires the strong cache keys of the build dependencies (and their runtime +dependencies).

+

The build dependencies of an element may have been updated since the artifact +was built. With the non-strict build plan the artifact will still be reused. +However, this means that we cannot use a strong cache key calculated purely +based on the element definitions. We need a cache key that matches the +environment at the time the artifact was built, not the current definitions.

+

The only way to get the correct strong cache key is by retrieving it from +the metadata stored in the artifact. As artifacts may need to be pulled from a +remote artifact cache, the strong cache key is not readily available early +in the build session. However, it can always be retrieved when an element is +about to be built, as the dependencies are guaranteed to be in the local +artifact cache at that point.

+

Element._get_cache_key_from_artifact() extracts the strong cache key +from an artifact in the local cache. Element._get_cache_key_for_build() +calculates the strong cache key that is used for a particular build job. +This is used for the embedded metadata and also as key to store the artifact in +the cache.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_caches.html b/arch_caches.html new file mode 100644 index 000000000..e2a0b61d8 --- /dev/null +++ b/arch_caches.html @@ -0,0 +1,174 @@ + + + + + + + + + Caches — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Caches

+

BuildStream uses local caches to avoid repeating work, and can have remote +caches configured to allow the results of work to be shared between multiple +users. There are caches for both elements and sources that map keys to relevant +metadata and point to data in CAS.

+
+

Content Addressable Storage (CAS)

+

The majority of data is stored in Content Addressable Storage or CAS, which +indexes stored files by the SHA256 hash of their contents. This allows for a +flat file structure as well as any repeated data to be shared across a CAS. In +order to store directory structures BuildStream’s CAS uses protocol buffers +for storing directory and file information as defined in Googles REAPI.

+

The data itself is stored in CAS which is defined by the remote execution protocol, +and BuildStream also uses the remote asset protocol in order to address stored +content using symbolic labesl, such as artifact names for +artifacts.

+
+
+

Artifact caches

+

Artifacts store build results of an element which is then referred to by its +cache key (described in Cache keys). The artifacts information is then +stored in a protocol buffer, defined in artifact.proto, which includes +metadata such as the digest of the files root; strong and weak keys; and log +files digests. The digests point to locations in the CAS of relavant files and +directories, allowing BuildStream to query remote CAS servers for this +information.

+
+
+

Source caches

+

Sources are cached by running the Source.stage method and capturing the directory output of +this into the CAS, which then use the sources key to refer to this. The source +key will be calculated with the plugins defined Plugin.get_unique_key and, depending on whether the source +requires previous sources to be staged (e.g. the patch plugin), the unique key +of all sources listed before it in an element. Source caches are simpler than +artifacts, as they just need to map a source key to a directory digest, with no +additional metadata.

+
+

Note

+

Not all plugins use the same result as the staged output for workspaces. As a +result when initialising a workspace, BuildStream may require fetching the +original source if it only has the source in the source cache.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_data_model.html b/arch_data_model.html new file mode 100644 index 000000000..380ee73aa --- /dev/null +++ b/arch_data_model.html @@ -0,0 +1,264 @@ + + + + + + + + + Data model — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Data model

+

This section details the data model on which the BuildStream core operates. This +includes an overview of the project data model which is BuildStream’s main input, +the user preferences, and local state.

+
+

Project

+

The Project object is the main component of a given BuildStream project, and +is responsible for loading and validating the project.conf, and +providing this loaded project data in a convenient way to the BuildStream core.

+

Conceptually, the project is a container for the Elements, +which are declared within a user’s project, and as such acts as a factory for instantiating +elements at load time.

+
+
+

Element

+

Elements are the main processing unit in a pipeline. These +are the loaded representation of the .bst files loaded from the project’s element path.

+

The Element is an abstract base class which cannot do anything on its own, its +concrete class is defined by plugins which are either included in the BuildStream +core set of plugins or loaded from external sources defined by the project.

+

The responsibilities of an element include:

+
    +

  • Loading the element’s configuration from the core provided dictionary.

    • +

  • Providing a unique key for any element specific configuration which might +affect the output produced by the element.

    • +

  • Configuring the sandbox.

    • +

  • Staging the data into the sandbox, which might include Sources and +the outputs of previous elements.

    • +

  • Assembling the output artifact.

    • +
+
+

Element data structure

+

The properties of an element are a composition of what the BuildStream core understands, +the configurations exposed by the Element plugin, and free form data which allows +annotations and configurations which can be read back by reverse dependencies during +processing, as illustrated here:

+_images/arch-datamodel-element.svg +
+
+

Element composition

+

The element is composed of configurations which are sourced from various entry +points using the low level YAML utilities.

+

This composition takes place after includes and +conditional directives are processed, while +list composition directives are processed +as a result of this composition.

+

Here is a diagram showing which sources take precedence in the composition process +which results in the final element configuration being resolved:

+_images/arch-datamodel-element-composition.svg +

Note that not all BuildStream Core Data is understood by the Element, but a great +deal of configurations understood by the Element is also understood by the core and +has default configurations built into BuildStream and configurable with the project +configuration. These include values such as variables, environment, sandbox, etc.

+

As shown above, composition is performed in two stages for each element. First +we compose everything below the line, this happens just once per ‘kind’ of +element - the result is re-used. Secondly, we compose the element declaration +on top.

+
+
+
+

Source

+

Sources are the abstract objects which are responsible +for obtaining remote source code or data to import into the build environment, and +ensuring that it is done in a bit-for-bit reproducible way without any contamination +of the host or build environment.

+

This is to say that:

+
    +

  • User configuration on the host, or filesystem outside of BuildStream designated +directories, must never be modified as a side-effect of running BuildStream.

    • +

  • When the Source uses host tools, host side configurations must never result in +deviations of what is staged to a build directory. The Source must behave exactly +the same way regardless of host side configurations.

    • +
+

The responsibilities of a source include:

+
    +

  • Loading the source’s configuration from the core provided dictionary.

    • +

  • Providing a unique key for any source specific configuration which might +affect the staged source.

    • +

  • Implement discovery of new versions of the source upstream (referred to as “tracking”).

    • +

  • Staging the unpacked source to a given directory.

    • +

  • Preparing workspaces.

    • +
+
+

Source data structure

+

Similar to the Element, the properties of a source are a composition of what +the BuildStream core understands and the configurations exposed by the Source +plugin:

+_images/arch-datamodel-source.svg +
+

Note

+

In .bst files, the BuildStream core configurations and Source specific configurations +share the same dictionary.

+

Strictly speaking this is limiting, but provides a measure of convenience as .bst +files are a bit less wordy to express.

+
+
+
+

Source composition

+

Source composition is much simpler than Element composition, because defaults +cannot be specified at the project level, excepting for Source type specific +value overrides.

+_images/arch-datamodel-source-composition.svg +
+
+
+

Context

+

The Context object is a very centric part of the BuildStream data model, and is +not a part of the Project data described above but rather is where we load and +store all of the user preferences.

+

User preferences are sourced from various locations, but usually have a default, +an option in the user configuration file, and an option to override it on the +command line.

+_images/arch-datamodel-context.svg +

Asides from being a focal point for loading and storing all user configuration, +the Context object also plays a central role in the logging framework.

+
+
+

Workspaces

+

The Workspaces object is yet another kind of state. Unlike the Context and +the Project data model, the Workspaces object loads, saves and stores in +memory the local state regarding a user’s active and open workspaces.

+

These are stored in the local state .bst/ subdirectory of users projects.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_dependency_model.html b/arch_dependency_model.html new file mode 100644 index 000000000..777b37b08 --- /dev/null +++ b/arch_dependency_model.html @@ -0,0 +1,183 @@ + + + + + + + + + Dependency model — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Dependency model

+

Elements in the data model are related by their dependencies. In BuildStream, there +are two types of relationship that an Element may have with a dependency, +build and runtime dependencies. More often than not, +an element will require its dependency both to build and also at runtime.

+

Consider a simple build scenario where you want to build an application, which +requires a service be present in order to function properly at runtime, a +compiler that need only be present at build time, and a runtime environment +or base system which must be required all the time:

+_images/arch-dependency-model.svg +

Note that in BuildStream we are only concerned with element level granularity +in our dependency model, and there is no way to depend on only a part of an element’s +output artifact. Instead we can employ compose and +filter elements in conjunction with split rules +to achieve sub artifact granularity at build and deploy time.

+

When developing BuildStream, it is important to understand the distinction +between dependency types and element Scope, +which acts as a selector of which elements to consider in the dependency +graph of a given element when performing recursive activities.

+
+

Scope

+
    +

  • Scope.ALL

    +

    In the Scope.ALL scope, all elements +are considered.

    +

    This is used in some cases to forcefully fetch, pull or build all dependencies +of a given element, even when not all of them are needed.

    +

    This scope simply includes all of the dependencies, including the element itself.

    +
    • +

  • Scope.RUN

    +

    In the Scope.RUN scope, only elements +which are required to run are considered, including the element itself. Note +that these are transitive - the service also requires the base runtime.

    +

    This is used when for example, launching a bst shell environment +for the purpose of running, or in any case we need to consider which +elements are required to run.

    +_images/arch-dependency-model-runtime.svg +
    • +

  • Scope.BUILD

    +

    In the Scope.BUILD scope, only +elements which are required to build are considered, excluding the +element we intend to build.

    +_images/arch-dependency-model-build.svg +

    Note that build type dependencies are not transitive, which is why the +Bootstrap element is not selected when pulling in the Compiler to +build the Application.

    +

    Further, note that we still follow the Compiler dependency on the +Base Runtime, this is because when we depend on an element for the +purpose of building, we expect that element to run and as such +we include all of the runtime dependencies of build dependencies +when selecting the Scope.BUILD elements.

    +
    • +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_overview.html b/arch_overview.html new file mode 100644 index 000000000..043f5858d --- /dev/null +++ b/arch_overview.html @@ -0,0 +1,130 @@ + + + + + + + + + Overview of modules — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Overview of modules

+

Below is a basic overview of the modules, what they are for, and generally +what their stacking order is internally in BuildStream.

+_images/arch-overview.svg +
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_program_flow.html b/arch_program_flow.html new file mode 100644 index 000000000..a765bec37 --- /dev/null +++ b/arch_program_flow.html @@ -0,0 +1,132 @@ + + + + + + + + + Overview of program flow — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Overview of program flow

+

Here is a little chart to show the approximate highlevel program flow of +BuildStream. This is in no way a complete flow chart of BuildStream, but +should provide some highlevel insight into how the program operates in +general.

+_images/arch-program-flow.svg +
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_remote_execution.html b/arch_remote_execution.html new file mode 100644 index 000000000..a8e621cc9 --- /dev/null +++ b/arch_remote_execution.html @@ -0,0 +1,158 @@ + + + + + + + + + Remote execution — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Remote execution

+

Remote execution allows builds to take place on different machines from the machine bst is run on, allowing faster builds, shared resources and different build capabilities, for example different machine architectures.

+
+

Sandbox extension

+

The previous section Sandboxing describes the two forms of local sandbox: the bubblewrap sandbox and the less common chroot sandbox (which is used on non-Linux POSIX environments). Remote execution uses a third type, the remote sandbox, which functions similarly to the local sandbox types, but is responsible for causing the build to occur on a remote system. Remote sandboxes should produce the same result as local sandboxes.

+
+
+

Artifact caches and other storage

+

BuildStream can transmit the results of local builds to remote artifact caches and retrieve them later. The same mechanism is used for remote execution. The filesystem staged before building is stored in a local content-addressable store, which may be the same as the local artifact cache. The command to execute is also stored as an object in local CAS. Both the initial source filesystem and command are transmitted to remote storage specific to the remote execution service, and after the build is complete, the filesystem after build is retrieved from remote storage to the local CAS. The remote execution service uses the same communication protocol as artifact caches, and may use the same internal storage, but may not implement the Remote Asset API used by BuildStream to store full artifacts.

+_images/arch-remote-execution.svg +

After sending the prerequisite file system and command to the remote execution cache, BuildStream uses the Remote Execution API (REAPI) [1] to signal to the build server that it should perform a build. +How the build service does this is not BuildStream’s concern, but typically there will be a worker that will be assigned the work, retrieve the inputs of the build from CAS, carry out the build, and store the results in CAS. The remote execution service is entitled to return a cached result, if the filesystem and command have already been executed.

+

BuildStream will continue to poll the remote execution server until the build is completed or lost. If it’s completed (successfully or otherwise) the resulting objects (typically the finished file system and logs of stdout and stderr) will be pulled to the local cache. BuildStream will retry jobs that are lost by the remote build server or which complete with certain error types.

+

sandboxremote.py contains all the communication with the remote execution API.

+

After a successful build, BuildStream will push the completed artifact to the remote artifact servers as if it had built it locally.

+
+
+

Use of sandboxes outside builds

+

Because staging requires integration-commands to be run, a sandbox is necessary for the bst artifact checkout command to run.

+

A sandbox is also required for the bst shell command. Because the REAPI does not provide any mechanism for interactive use, bst shell will always use a local sandbox.

+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_sandboxing.html b/arch_sandboxing.html new file mode 100644 index 000000000..90e01dbd5 --- /dev/null +++ b/arch_sandboxing.html @@ -0,0 +1,295 @@ + + + + + + + + + Sandboxing — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Sandboxing

+
+

Introduction

+

BuildStream assembles each element in a sandbox. The sandbox is a container +environment which serves two purposes: giving BuildStream control over +all build aspects in order to ensure reproducibility of build results, +and providing safety guarantees for the host system that BuildStream is +running on.

+

The exact implementation of the sandbox varies depending on which platform you +are running BuildStream. See below for backend-specific details.

+

There are several factors that affect the build output and must therefore be +under BuildStream’s control:

+
    +

  • Filesystem contents and metadata

    • +

  • The user and permissions model

    • +

  • Network access

    • +

  • Device access

    • +
+

Each of these is detailed below.

+

For safety reasons, BuildStream also controls the following things:

+
    +

  • Access to files outside of the sandbox directory

    • +

  • Access to certain kernel-specific syscalls

    • +
+

Creating a sandbox can require special priviliges. This is a safety concern too +because bugs in the bst program can cause damage to a host if the program is +running with extra privileges. The exact priviliges that are required depend on +your platform and backend.

+

Element plugins can run arbitary commands within the sandbox using the +sandbox API.

+
+
+

What elements can and can’t do in the sandbox

+

This section specifies how BuildStream sandboxes are intended to work. A +specific sandbox provider may not necessarily be able to achieve all of the +requirements listed below so be sure to read the “platform notes” section as +well.

+
+

Filesystem access

+

The filesystem inside sandboxes should be read-only during element assembly, +except for certain directories which element plugins can mark as being +read/write. Most elements plugins derive from BuildElement, which marks %{build-root} and +%{install-root} as read/write.

+

When running integration commands or bst shell, the sandbox should have a +fully read-write filesystem. The changes made here do not need to persist +beyond the lifetime of that sandbox, and must not affect the contents of +artifacts stored in the cache.

+

Certain top level directories should be treated specially in all sandboxes:

+
    +

  • The /dev directory should contain device nodes, which are described in +a separate section.

    • +

  • The /proc directory should have a UNIX ‘procfs’ style filesystem mounted. +It should not expose any information about processes running outside of the +sandbox.

    • +

  • The /tmp directory should be writable.

    • +
+
+
+

Filesystem metadata

+

The writable areas inside a BuildStream sandbox are limited in what metadata +can be written and stored.

+
    +

  • All files must be owned by UID 0 and GID 0

    • +

  • No files may have the setuid or setgid bits set

    • +

  • Extended file attributes (xattrs) cannot be written to or read.

    • +

  • Hardlinks to other files can be created, but the information about which +files are hardlinked to each other will not be stored in the artifact +that is created from the sandbox.

    • +
+

These restrictions are due to technical limitations. In future we hope to +support a wider range of filesystem metadata operations. See issue #38 for more details.

+
+
+

User and permissions model

+

All commands inside the sandbox run with user ID 0 and group ID 0. It should +not be possible to become any other user ID.

+
+
+

Network access

+

Builds should not be able to access the network at all from the sandbox. All +remote resources needed to build an element must be specified in the element’s +sources list so that BuildStream is able to see when they have changed.

+

A sandbox opened by bst shell should allow network access.

+
+
+

Device access

+

Builds should not be able to access any hardware devices at all.

+

A few standard UNIX device files are needed, the whitelist is:

+
    +

  • /dev/full

    • +

  • /dev/null

    • +

  • /dev/urandom

    • +

  • /dev/random

    • +

  • /dev/zero

    • +
+

It may seem odd that we have sources of randomness in the sandbox, but a lot of +tools do expect them to exist. We take the view that it’s up to integrators to +ensure that elements do not deliberately include randomness in their output.

+

A sandbox opened by bst shell can make any devices available. There needs to +be a console device so that it can be used interactively.

+
+
+
+

Platform notes

+

BuildStream delegates sandboxing for local builds to the buildbox-run +command. buildbox-run provides a platform-independent interface to execute +commands in a sandbox based on parts of the Remote Execution API.

+
+

Linux

+

The recommended buildbox-run implementation for Linux is +buildbox-run-bubblewrap, in combination with buildbox-fuse.

+

These implementations use the following isolation and sandboxing primitives:

+
    +

  • bind mounts

    • +

  • FUSE

    • +

  • Mount namespaces

    • +

  • Network namespaces

    • +

  • PID (process ID) namespaces

    • +

  • User namespaces (if available)

    • +

  • seccomp

    • +
+

We access all of these features through a sandboxing tool named Bubblewrap.

+

User namespaces are not enabled by default in all Linux distributions. +BuildStream still runs on such systems but can’t build projects that set +build-uid or build-gid in the sandbox configuration.

+

The Linux platform can operate as a standard user, if unprivileged user namespace +support is available. If user namespace support is not available you have the +option of installing bubblewrap as a setuid binary to avoid needing to run the +entire bst process as the root user.

+

FUSE is used to provide access to directories and files stored in CAS without +having to copy or hardlink the complete input tree into a regular filesystem +directory structure for each build job.

+

Some of the operations on filesystem metadata listed above are not prohibited +by the sandbox, but will instead be silently dropped when an artifact is +created. For more details see issue #38.

+

Some details of the host machine are currently leaked by this platform backend. +For more details, see issue #262.

+
+
+

Other POSIX systems

+

On other POSIX systems buildbox-run-userchroot may be used for sandboxing. +userchroot allows regular +users to invoke processes in a chroot environment.

+

buildbox-run-userchroot stages the input tree for each build job using +hardlinks to avoid more expensive file copies. To avoid cache corruption it is +vital that hardlinked files cannot be overwritten. Due to this it’s required +to run buildbox-casd as a separate user, which owns the files in the local +cache.

+

Network access is not blocked in the chroot. However since there is unlikely +to be a correct /etc/resolv.conf file, any network access that depends on +name resolution will most likely fail anyway.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/arch_scheduler.html b/arch_scheduler.html new file mode 100644 index 000000000..63e271489 --- /dev/null +++ b/arch_scheduler.html @@ -0,0 +1,233 @@ + + + + + + + + + Scheduler — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Scheduler

+

The Scheduler is what is responsible for running the main event loop and +dispatching Jobs to complete tasks on behalf of Queues.

+
+

Jobs

+

The basic functionality of multiprocessing tasks is implemented by the base Job +class, which is derived in a few ways but for now we’ll only talk about the +ElementJob type since that is the most centric.

+

The Job class has the following responsibilities:

+
    +

  • Starting the given job as a subprocess.

    • +

  • Offering an abstract method for subclasses to handle the outcome of +a job when it completes.

    • +

  • Forcefully terminating its subprocess.

    • +

  • Suspending and resuming its subprocess.

    • +

  • Declaring the types of resources it will require, and which resources +it will require exclusively.

    • +
+

Below is a rough outline of the interactions between the main process +and job specific child process:

+_images/arch-scheduler-job.svg +
+
+

Resources

+

To understand how we manage load balancing in the scheduler it is important +to understand resources.

+

For the scheduler, resources are domains which a Job can request which represent +physical resources, such as the CPU or some network bandwidth, or the local +artifact cache.

+

This is used by the Scheduler when consuming Jobs from Queues and deciding +how many jobs can be run at a given time.

+
+
+

Queues

+

The various Queue implementations in the Scheduler can be combined such that +parallelism is maximized. For example one can Track and Build inside the +same session, in this way one does not need to wait for a tracking session to +complete in order to start building.

+

The input elements to the scheduler are expected to be sorted in depth first +order whenever the order is important, again allowing maximum parallelism +at build time.

+_images/arch-scheduler-queues.svg +

The Queue implementations are:

+
    +

  • Track

    +

    The tracking queue must always come first if it is used in the given session. +This is because the Source “ref”, and consequently the cache key of any elements +which have been requested for tracking, cannot be known until tracking is complete.

    +
    • +

  • Pull

    +

    The pull queue tries to obtain a built artifact from a remote artifact server, +it should be placed in advance of the fetch queue if used, since we can safely +avoid fetching if fetching is not imperative, and we already have a cached +artifact.

    +
    • +

  • Fetch

    +

    The fetch queue attempts to download source code to build the given element, +this activity is sometimes skipped if the artifact is already present, or +if the exact source code is already present.

    +
    • +

  • Build

    +

    The build queue attempts to build the element if its artifact is not locally +present.

    +
    • +

  • Push

    +

    The push queue attempts to push the resulting artifact to a remote artifact +server.

    +
    • +
+
+
+

Queue internals

+

Internally, the queue has an input queue and an output queue.

+_images/arch-scheduler-queue-ports.svg +

When elements are on the input queue they get queried for their QueueStatus +in order to determine which elements should be processed or moved from the input +queue to the output queue. When elements are on the output queue, they are ready +to be consumed by the scheduler and moved on to the next queue; however each +queue holds on to the result status of every element which passed through for later +reference and reports to the user.

+
+
+

Scheduler

+

The scheduler itself has the main responsibility of popping off jobs from +the existing queues it was given, and running the jobs as long as elements +remain to be processed.

+

A huge simplification of this can be visualized as follows:

+_images/arch-scheduler-run.svg +

This is implemented by iterating over the Queues given to the scheduler +and processing any “Ready” elements so long as there are sufficient resource +tokens available for the ready elements to run, and by moving the “Done” +elements onto the subsequent queues in the list of queues.

+
+

Note

+

When looking for “Ready” elements in the queues, we iterate over the +queue list in reverse order. This is important to allow elements to +get as far as they can in the queue list as fast as possible, and helps +to prevent resource starvation.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.buildelement.html b/buildstream.buildelement.html new file mode 100644 index 000000000..dd563ca3d --- /dev/null +++ b/buildstream.buildelement.html @@ -0,0 +1,524 @@ + + + + + + + + + BuildElement - Abstract class for build elements — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

BuildElement - Abstract class for build elements

+

The BuildElement class is a convenience element one can derive from for +implementing the most common case of element.

+
+

Built-in functionality

+

The BuildElement base class provides built in functionality that could be +overridden by the individual plugins.

+

This section will give a brief summary of how some of the common features work, +some of them or the variables they use will be further detailed in the following +sections.

+
+

The strip-binaries variable

+

The strip-binaries variable is by default empty. You need to use the +appropiate commands depending of the system you are building. +If you are targetting Linux, ones known to work are the ones used by the +freedesktop-sdk, you can take a look to them in their +project.conf

+
+
+

Location for staging dependencies

+

The BuildElement supports the “location” dependency configuration, +which means you can use this configuration for any BuildElement class.

+

The “location” configuration defines where the dependency will be staged in the +build sandbox.

+

Example:

+

Here is an example of how one might stage some dependencies into +an alternative location while staging some elements in the sandbox root.

+
# Stage these build dependencies in /opt
+#
+build-depends:
+- baseproject.bst:opt-dependencies.bst
+  config:
+    location: /opt
+
+# Stage these tools in "/" and require them as
+# runtime dependencies.
+depends:
+- baseproject.bst:base-tools.bst
+
+
+
+

Note

+

The order of dependencies specified is not significant.

+

The staging locations will be sorted so that elements are staged in parent +directories before subdirectories.

+
+
+
+

Location for running commands

+

The command-subdir variable sets where commands will be executed, +and the directory will be created automatically if it does not exist.

+

The command-subdir is a relative path from %{build-root}, and +cannot be a parent or adjacent directory, it must expand to a subdirectory +of ${build-root}.

+
+
+

Location for configuring the project

+

The conf-root is the location that specific build elements can use to look for build configuration files. +This is used by elements such as autotools, +cmake, +meson, +setuptools and +pip.

+

The default value of conf-root is defined by default as .. This means that if +the conf-root is not explicitly set to another directory, the configuration +files are expected to be found in command-subdir.

+
+

Separating source and build directories

+

A typical example of using conf-root is when performing +autotools builds +where your source directory is separate from your build directory.

+

This can be achieved in build elements which use conf-root as follows:

+
variables:
+  # Specify that build configuration scripts are found in %{build-root}
+  conf-root: "%{build-root}"
+
+  # The build will run in the `_build` subdirectory
+  command-subdir: _build
+
+
+
+
+
+

Install Location

+

Build elements must install the build output to the directory defined by install-root.

+

You need not set or change the install-root variable as it will be defined +automatically on your behalf, and it is used to collect build output when creating +the resulting artifacts.

+

It is important to know about install-root in order to write your own +custom install instructions, for example the +cmake +element will use it to specify the DESTDIR.

+
+
+
+

Abstract method implementations

+
+

Element.configure_sandbox()

+

In Element.configure_sandbox(), +the BuildElement will ensure that the sandbox locations described by the %{build-root} +and %{install-root} variables are marked and will be mounted read-write for the +assemble phase.

+

The working directory for the sandbox will be configured to be the %{build-root}, +unless the %{command-subdir} variable is specified for the element in question, +in which case the working directory will be configured as %{build-root}/%{command-subdir}.

+
+
+

Element.stage()

+

In Element.stage(), the BuildElement +will do the following operations:

+
    +

  • Stage all of the build dependencies into the sandbox root.

    • +

  • Run the integration commands for all staged dependencies using +Element.integrate()

    • +

  • Stage any Source on the given element to the %{build-root} location +inside the sandbox, using +Element.stage_sources()

    • +
+
+
+

Element.assemble()

+

In Element.assemble(), the +BuildElement will proceed to run sandboxed commands which are expected to be +found in the element configuration.

+

Commands are run in the following order:

+
    +

  • configure-commands: Commands to configure the build scripts

    • +

  • build-commands: Commands to build the element

    • +

  • install-commands: Commands to install the results into %{install-root}

    • +

  • strip-commands: Commands to strip debugging symbols installed binaries

    • +
+

The result of the build is expected to end up in %{install-root}, and +as such; Element.assemble() method will return the %{install-root} for +artifact collection purposes.

+
+

Note

+

In the case that the element is currently workspaced, the configure-commands +will only be run in subsequent builds until they succeed at least once, unless +bst workspace reset –soft is called on the +workspace to explicitly avoid an incremental build.

+
+
+
+
+
+
+class BuildElement(context: Context, project: Project, load_element: LoadElement, plugin_conf: str | None, *, artifact_key: str = None)
+

Bases: Element

+
+
+configure(node)
+

Configure the Plugin from loaded configuration data

+
+
Parameters:
+

node – The loaded configuration dictionary

+
+
Raises:
+
    +

  • .SourceError – If it’s a Source implementation

    • +

  • .ElementError – If it’s an Element implementation

    • +
+
+
+

Plugin implementors should implement this method to read configuration +data and store it.

+

The MappingNode.validate_keys() method +should be used to ensure that the user has not specified keys in node which are unsupported +by the plugin.

+
+ +
+
+configure_dependencies(dependencies)
+

Configure the Element with regards to it’s build dependencies

+

Elements can use this method to parse custom configuration which define their +relationship to their build dependencies.

+

If this method is implemented, then it will be called with all direct build dependencies +specified in their element declaration in a list.

+

If the dependency was declared with custom configuration, it will be provided along +with the dependency element, otherwise None will be passed with dependencies which +do not have any additional configuration.

+

If the user has specified the same build dependency multiple times with differing +configurations, then those build dependencies will be provided multiple times +in the dependencies list.

+
+
Parameters:
+

dependencies (list) – A list of DependencyConfiguration +objects

+
+
Raises:
+

.ElementError – When the element raises an error

+
+
+

The format of the MappingNode provided as +DependencyConfiguration.config +method should be called by the implementing plugin in order to validate it.

+
+

Note

+

It is unnecessary to implement this method if the plugin does not support +any custom dependency configuration.

+
+
+ +
+
+preflight()
+

Preflight Check

+
+
Raises:
+
    +

  • .SourceError – If it’s a Source implementation

    • +

  • .ElementError – If it’s an Element implementation

    • +
+
+
+

This method is run after Plugin.configure() +and after the pipeline is fully constructed.

+

Implementors should simply raise SourceError or ElementError +with an informative message in the case that the host environment is +unsuitable for operation.

+

Plugins which require host tools (only sources usually) should obtain +them with utils.get_host_tool() which +will raise an error automatically informing the user that a host tool is needed.

+
+ +
+
+get_unique_key()
+

Return something which uniquely identifies the plugin input

+
+
Returns:
+

A string, list or dictionary which uniquely identifies the input

+
+
+

This is used to construct unique cache keys for elements +and sources, sources should return something which uniquely identifies the payload, +such as an sha256 sum of a tarball content.

+

Elements and Sources should implement this by collecting any configurations +which could possibly affect the output and return a dictionary of these settings.

+

For Sources, this is guaranteed to only be called if +Source.is_resolved() has returned +True which is to say that the Source is expected to have an exact +source ref indicating exactly what source is going to be staged.

+
+

Note

+

If your plugin is concerned with API stability, then future extensions of your +plugin YAML configuration which affect the unique key returned here should be added +to this key with care.

+

A good rule of thumb is to only compute the new value in the returned key if +the value of the newly added YAML key is not equal to it’s default value.

+
+
+ +
+
+configure_sandbox(sandbox)
+

Configures the the sandbox for execution

+
+
Parameters:
+

sandbox – The build sandbox

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to configure the sandbox object +for execution.

+
+ +
+
+stage(sandbox)
+

Stage inputs into the sandbox directories

+
+
Parameters:
+

sandbox – The build sandbox

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to populate the sandbox +directory with data. This is done either by staging Source +objects, by staging the artifacts of the elements this element depends +on, or both.

+
+ +
+
+assemble(sandbox)
+

Assemble the output artifact

+
+
Parameters:
+

sandbox – The build sandbox

+
+
Returns:
+

An absolute path within the sandbox to collect the artifact from

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to create an output +artifact from its sources and dependencies.

+
+ +
+
+generate_script()
+

Generate a build (sh) script to build this element

+
+
Returns:
+

A string containing the shell commands required to build the element

+
+
+

BuildStream guarantees the following environment when the +generated script is run:

+
    +

  • All element variables have been exported.

    • +

  • The cwd is self.get_variable(‘build-root’)/self.normal_name.

    • +

  • $PREFIX is set to self.get_variable(‘install-root’).

    • +

  • The directory indicated by $PREFIX is an empty directory.

    • +
+

Files are expected to be installed to $PREFIX.

+

If the script fails, it is expected to return with an exit +code != 0.

+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.downloadablefilesource.html b/buildstream.downloadablefilesource.html new file mode 100644 index 000000000..3090677d5 --- /dev/null +++ b/buildstream.downloadablefilesource.html @@ -0,0 +1,463 @@ + + + + + + + + + DownloadableFileSource - Abstract class for sources downloaded from a URI — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

DownloadableFileSource - Abstract class for sources downloaded from a URI

+

This DownloadableFileSource class is a convenience class on can derive for +implementing sources that get downloaded from a URI.

+

It provides utilities around handling mirrors, tracking and fetching the source.

+

Any derived classes must write their own stage() and get_unique_key() +implementation.

+
+

SourceMirror extra data “http-auth”

+

The DownloadableFileSource, and consequently any Source +implementations which derive from DownloadableFileSource, support the “http-auth” +extra data returned by SourceMirror plugins +through Source.translate_url().

+

This functionality is available Since: 2.2.

+

This allows one to use SourceMirror plugins +to add an authorization header to the GET requests.

+

Example:

+
class MySourceMirror(SourceMirror):
+
+     def translate_url(
+         self,
+         *,
+         alias: str,
+         alias_url: str,
+         source_url: str,
+         extra_data: Optional[Dict[str, Any]],
+     ) -> str:
+
+         #
+         # Set the "http-auth" extra data
+         #
+         if extra_data is not None:
+             extra_data["http-auth"] = "bearer"
+
+         # ...
+
+
+

Only the “http-auth” value bearer is supported.

+

Example:

+

If the URL reported by SourceMirror.translate_url() +is http://flying-ponies.com/downloads/pony.tgz, then a corresponding entry will be expected in the +user’s ~/.netrc:

+
flying-ponies.com
+    password 1234
+
+
+

DownloadableFileSource will add the following header to the GET request to download the file:

+
Authorization: Bearer 1234
+
+
+
+
+
+
+class DownloadableFileSource(context: Context, project: Project, meta: MetaSource, variables: Variables, *, alias_override: Tuple[str, AliasSubstitution] | None = None, unique_id: int | None = None)
+

Bases: Source

+
+
+COMMON_CONFIG_KEYS = ['kind', 'directory', 'url', 'ref']
+

Common source config keys

+

Source config keys that must not be accessed in configure(), and +should be checked for using node.validate_keys().

+
+ +
+
+configure(node)
+

Configure the Plugin from loaded configuration data

+
+
Parameters:
+

node – The loaded configuration dictionary

+
+
Raises:
+
    +

  • .SourceError – If it’s a Source implementation

    • +

  • .ElementError – If it’s an Element implementation

    • +
+
+
+

Plugin implementors should implement this method to read configuration +data and store it.

+

The MappingNode.validate_keys() method +should be used to ensure that the user has not specified keys in node which are unsupported +by the plugin.

+
+ +
+
+preflight()
+

Preflight Check

+
+
Raises:
+
    +

  • .SourceError – If it’s a Source implementation

    • +

  • .ElementError – If it’s an Element implementation

    • +
+
+
+

This method is run after Plugin.configure() +and after the pipeline is fully constructed.

+

Implementors should simply raise SourceError or ElementError +with an informative message in the case that the host environment is +unsuitable for operation.

+

Plugins which require host tools (only sources usually) should obtain +them with utils.get_host_tool() which +will raise an error automatically informing the user that a host tool is needed.

+
+ +
+
+get_unique_key()
+

Return something which uniquely identifies the plugin input

+
+
Returns:
+

A string, list or dictionary which uniquely identifies the input

+
+
+

This is used to construct unique cache keys for elements +and sources, sources should return something which uniquely identifies the payload, +such as an sha256 sum of a tarball content.

+

Elements and Sources should implement this by collecting any configurations +which could possibly affect the output and return a dictionary of these settings.

+

For Sources, this is guaranteed to only be called if +Source.is_resolved() has returned +True which is to say that the Source is expected to have an exact +source ref indicating exactly what source is going to be staged.

+
+

Note

+

If your plugin is concerned with API stability, then future extensions of your +plugin YAML configuration which affect the unique key returned here should be added +to this key with care.

+

A good rule of thumb is to only compute the new value in the returned key if +the value of the newly added YAML key is not equal to it’s default value.

+
+
+ +
+
+is_cached() → bool
+

Get whether the source has a local copy of its data.

+

This method is guaranteed to only be called whenever +Source.is_resolved() +returns True.

+

Returns: whether the source is cached locally or not.

+
+ +
+
+load_ref(node)
+

Loads the SourceRef for this Source from the specified node.

+
+
Parameters:
+

node – The YAML node to load the ref from

+
+
+

Working with the source ref is discussed here.

+
+

Note

+

The SourceRef for the Source is expected to be read at +Plugin.configure() time, +this will only be used for loading refs from alternative locations +than in the element.bst file where the given Source object has +been declared.

+
+
+ +
+
+get_ref()
+

Fetch the SourceRef

+
+
Returns:
+

The internal SourceRef, or None

+
+
+

Working with the source ref is discussed here.

+
+ +
+
+set_ref(ref, node)
+

Applies the internal ref, however it is represented

+
+
Parameters:
+
+
+
+

The implementor must update the node parameter to reflect the new ref, +and it should store the passed ref so that it will be returned in any +later calls to Source.get_ref().

+

The passed ref parameter is guaranteed to either be a value which has +been previously retrieved by the Source.get_ref() +method on the same plugin, or None.

+

Example:

+
# Implementation of Source.set_ref()
+#
+def set_ref(self, ref, node):
+
+    # Update internal state of the ref
+    self.ref = ref
+
+    # Update the passed node so that we will read the new ref
+    # next time this source plugin is configured with this node.
+    #
+    node["ref"] = self.ref
+
+
+

Working with the source ref is discussed here.

+
+ +
+
+track()
+

Resolve a new ref from the plugin’s track option

+
+
Parameters:
+

previous_sources_dir (str) – directory where previous sources are staged. +Note that this keyword argument is available only when +BST_REQUIRES_PREVIOUS_SOURCES_TRACK +is set to True.

+
+
Returns:
+

A new SourceRef, or None

+
+
+

If the backend in question supports resolving references from +a symbolic tracking branch or tag, then this should be implemented +to perform this task on behalf of bst source track +commands.

+

This usually requires fetching new content from a remote origin +to see if a new ref has appeared for your branch or tag. If the +backend store allows one to query for a new ref from a symbolic +tracking data without downloading then that is desirable.

+

Working with the source ref is discussed here.

+
+ +
+
+fetch()
+

Fetch remote sources and mirror them locally, ensuring at least +that the specific reference is cached locally.

+
+
Parameters:
+

previous_sources_dir (str) – directory where previous sources are staged. +Note that this keyword argument is available only when +BST_REQUIRES_PREVIOUS_SOURCES_FETCH +is set to True.

+
+
Raises:
+

.SourceError –

+
+
+

Implementors should raise SourceError if the there is some +network error or if the source reference could not be matched.

+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.element.html b/buildstream.element.html new file mode 100644 index 000000000..b20216970 --- /dev/null +++ b/buildstream.element.html @@ -0,0 +1,805 @@ + + + + + + + + + Element - Base element class — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Element - Base element class

+
+

Abstract Methods

+

For loading and configuration purposes, Elements must implement the +Plugin base class abstract methods.

+
+

Build Phase

+

The following methods are the foundation of the element’s build +phase, they must be implemented by all Element classes, unless +explicitly stated otherwise.

+ +
+
+

Miscellaneous

+

Miscellaneous abstract methods also exist:

+
    +

  • Element.generate_script()

    +

    For the purpose of bst source checkout --include-build-scripts, an Element may optionally implement this.

    +
    • +
+
+
+
+

Class Reference

+
+
+
+
+exception ElementError(message: str, *, detail: str = None, reason: str = None, collect: str = None, temporary: bool = False)
+

Bases: BstError

+

This exception should be raised by Element implementations +to report errors to the user.

+
+
Parameters:
+
    +

  • message – The error message to report to the user

    • +

  • detail – A possibly multiline, more detailed error message

    • +

  • reason – An optional machine readable reason string, used for test cases

    • +

  • collect – An optional directory containing partial install contents

    • +

  • temporary – An indicator to whether the error may occur if the operation was run again.

    • +
+
+
+
+ +
+
+class DependencyConfiguration(element: Element, path: str, config: MappingNode | None)
+

Bases: object

+

An object representing the configuration of a dependency

+

This is used to provide dependency configurations for elements which implement +Element.configure_dependencies()

+
+
+element: Element
+

The dependency Element

+
+ +
+
+path: str
+

The path used to refer to this dependency

+
+ +
+
+config: MappingNode | None
+

The custom dependency configuration, or None +if no custom configuration was provided

+
+ +
+ +
+
+class Element
+

Bases: Plugin

+

Base Element class.

+

All elements derive from this class, this interface defines how +the core will be interacting with Elements.

+
+
+BST_ARTIFACT_VERSION = 0
+

The element plugin’s artifact version

+

Elements must first set this to 1 if they change their unique key +structure in a way that would produce a different key for the +same input, or introduce a change in the build output for the +same unique key. Further changes of this nature require bumping the +artifact version.

+
+ +
+
+BST_STRICT_REBUILD = False
+

Whether to rebuild this element in non strict mode if +any of the dependencies have changed.

+
+ +
+
+BST_FORBID_RDEPENDS = False
+

Whether to raise exceptions if an element has runtime dependencies.

+
+ +
+
+BST_FORBID_BDEPENDS = False
+

Whether to raise exceptions if an element has build dependencies.

+
+ +
+
+BST_FORBID_SOURCES = False
+

Whether to raise exceptions if an element has sources.

+
+ +
+
+BST_RUN_COMMANDS = True
+

Whether the element may run commands using Sandbox.run.

+
+ +
+
+BST_ELEMENT_HAS_ARTIFACT = True
+

Whether the element produces an artifact when built.

+
+ +
+
+project_name
+

The name of the owning project

+
+

Attention

+

Combining this attribute with Plugin.name +does not provide a unique identifier for an element within a project, this is because +multiple junction elements can be used specify the same +project as a subproject.

+
+
+ +
+
+normal_name
+

A normalized element name

+

This is the original element without path separators or +the extension, it’s used mainly for composing log file names +and creating directory names and such.

+
+ +
+
+configure_dependencies(dependencies: Iterable[DependencyConfiguration]) → None
+

Configure the Element with regards to it’s build dependencies

+

Elements can use this method to parse custom configuration which define their +relationship to their build dependencies.

+

If this method is implemented, then it will be called with all direct build dependencies +specified in their element declaration in a list.

+

If the dependency was declared with custom configuration, it will be provided along +with the dependency element, otherwise None will be passed with dependencies which +do not have any additional configuration.

+

If the user has specified the same build dependency multiple times with differing +configurations, then those build dependencies will be provided multiple times +in the dependencies list.

+
+
Parameters:
+

dependencies (list) – A list of DependencyConfiguration +objects

+
+
Raises:
+

.ElementError – When the element raises an error

+
+
+

The format of the MappingNode provided as +DependencyConfiguration.config +method should be called by the implementing plugin in order to validate it.

+
+

Note

+

It is unnecessary to implement this method if the plugin does not support +any custom dependency configuration.

+
+
+ +
+
+configure_sandbox(sandbox: Sandbox) → None
+

Configures the the sandbox for execution

+
+
Parameters:
+

sandbox – The build sandbox

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to configure the sandbox object +for execution.

+
+ +
+
+stage(sandbox: Sandbox) → None
+

Stage inputs into the sandbox directories

+
+
Parameters:
+

sandbox – The build sandbox

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to populate the sandbox +directory with data. This is done either by staging Source +objects, by staging the artifacts of the elements this element depends +on, or both.

+
+ +
+
+assemble(sandbox: Sandbox) → str
+

Assemble the output artifact

+
+
Parameters:
+

sandbox – The build sandbox

+
+
Returns:
+

An absolute path within the sandbox to collect the artifact from

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to create an output +artifact from its sources and dependencies.

+
+ +
+
+generate_script() → str
+

Generate a build (sh) script to build this element

+
+
Returns:
+

A string containing the shell commands required to build the element

+
+
+

BuildStream guarantees the following environment when the +generated script is run:

+
    +

  • All element variables have been exported.

    • +

  • The cwd is self.get_variable(‘build-root’)/self.normal_name.

    • +

  • $PREFIX is set to self.get_variable(‘install-root’).

    • +

  • The directory indicated by $PREFIX is an empty directory.

    • +
+

Files are expected to be installed to $PREFIX.

+

If the script fails, it is expected to return with an exit +code != 0.

+
+ +
+
+sources() → Iterator[Source]
+

A generator function to enumerate the element sources

+
+
Yields:
+

The sources of this element

+
+
+
+ +
+
+dependencies(selection: Sequence[Element] = None, *, recurse: bool = True) → Iterator[Element]
+

A generator function which yields the build dependencies of the given element.

+

This generator gives the Element access to all of the dependencies which it is has +access to at build time. As explained in the dependency type documentation, +this includes the direct build dependencies of the element being built, along with any +transient runtime dependencies of those build dependencies.

+

Subsets of the dependency graph can be selected using the selection argument,, which +must consist of dependencies of this element. If the selection argument is specified as +None, then the self element on which this is called is used as the selection.

+

If recurse is specified (the default), the full dependencies will be listed +in deterministic staging order, starting with the basemost elements. Otherwise, +if recurse is not specified then only the direct dependencies will be traversed.

+
+
Parameters:
+
    +

  • selection (Sequence[Element]) – A list of dependencies to select, or None

    • +

  • recurse (bool) – Whether to recurse

    • +
+
+
Yields:
+

The dependencies of the selection, in deterministic staging order

+
+
+
+ +
+
+search(name: str) → Element | None
+

Search for a dependency by name

+
+
Parameters:
+

name – The dependency to search for

+
+
Returns:
+

The dependency element, or None if not found.

+
+
+
+ +
+
+node_subst_vars(node: ScalarNode) → str
+

Replace any variables in the string contained in the node and returns it.

+

Warning: The method is deprecated and will get removed in the next version

+
+
Parameters:
+

node – A ScalarNode loaded from YAML

+
+
Returns:
+

The value with all variables replaced

+
+
Raises:
+

.LoadError – When the node doesn’t contain a string or a variable was not found.

+
+
+

Example:

+
# Expect a string 'name' in 'node', substituting any
+# variables in the returned string
+name = self.node_subst_vars(node.get_scalar('name'))
+
+
+
+ +
+
+node_subst_sequence_vars(node: SequenceNode[ScalarNode]) → List[str]
+

Substitute any variables in the given sequence

+

Warning: The method is deprecated and will get removed in the next version

+
+
Parameters:
+

node – A SequenceNode loaded from YAML

+
+
Returns:
+

The list with every variable replaced

+
+
Raises:
+

.LoadError –

+
+
+
+ +
+
+compute_manifest(*, include: List[str] | None = None, exclude: List[str] | None = None, orphans: bool = True) → str
+

Compute and return this element’s selective manifest

+

The manifest consists on the list of file paths in the +artifact. The files in the manifest are selected according to +include, exclude and orphans parameters. If include is +not specified then all files spoken for by any domain are +included unless explicitly excluded with an exclude domain.

+
+
Parameters:
+
    +

  • include – An optional list of domains to include files from

    • +

  • exclude – An optional list of domains to exclude files from

    • +

  • orphans – Whether to include files not spoken for by split domains

    • +
+
+
Yields:
+

The paths of the files in manifest

+
+
+
+ +
+
+get_artifact_name(key: str | None = None) → str
+

Compute and return this element’s full artifact name

+

Generate a full name for an artifact, including the project +namespace, element name and cache key.

+

This can also be used as a relative path safely, and +will normalize parts of the element name such that only +digits, letters and some select characters are allowed.

+
+
Parameters:
+

key – The element’s cache key. Defaults to None

+
+
Returns:
+

The relative path for the artifact

+
+
+
+ +
+
+stage_artifact(sandbox: Sandbox, *, path: str = None, action: str = <buildstream.types.OverlapAction object>, include: ~typing.List[str] | None = None, exclude: ~typing.List[str] | None = None, orphans: bool = True) → FileListResult
+

Stage this element’s output artifact in the sandbox

+

This will stage the files from the artifact to the sandbox at specified location. +The files are selected for staging according to the include, exclude and orphans +parameters; if include is not specified then all files spoken for by any domain +are included unless explicitly excluded with an exclude domain.

+
+
Parameters:
+
    +

  • sandbox – The build sandbox

    • +

  • path – An optional sandbox relative path

    • +

  • action (OverlapAction) – The action to take when overlapping with previous invocations

    • +

  • include – An optional list of domains to include files from

    • +

  • exclude – An optional list of domains to exclude files from

    • +

  • orphans – Whether to include files not spoken for by split domains

    • +
+
+
+

:raises (ElementError): If the element has not yet produced an artifact.

+
+
Returns:
+

The result describing what happened while staging

+
+
+
+

Note

+

Directories in dest are replaced with files from src, +unless the existing directory in dest is not empty in +which case the path will be reported in the return value.

+
+
+

Attention

+

When staging artifacts with their dependencies, use +Element.stage_dependency_artifacts() +instead.

+
+
+ +
+
+stage_dependency_artifacts(sandbox: Sandbox, selection: ~typing.Sequence[Element] = None, *, path: str = None, action: str = <buildstream.types.OverlapAction object>, include: ~typing.List[str] | None = None, exclude: ~typing.List[str] | None = None, orphans: bool = True) → None
+

Stage element dependencies in scope

+

This is primarily a convenience wrapper around +Element.stage_artifact() +which takes care of staging all the dependencies in staging order and issueing the +appropriate warnings.

+

The selection argument will behave in the same was as specified by +Element.dependencies(), +If the selection argument is specified as None, then the self element on which this +is called is used as the selection.

+
+
Parameters:
+
    +

  • sandbox (Sandbox) – The build sandbox

    • +

  • selection (Sequence[Element]) – A list of dependencies to select, or None

    • +

  • path (str) – An optional sandbox relative path

    • +

  • action (OverlapAction) – The action to take when overlapping with previous invocations

    • +

  • include (List[str]) – An optional list of domains to include files from

    • +

  • exclude (List[str]) – An optional list of domains to exclude files from

    • +

  • orphans (bool) – Whether to include files not spoken for by split domains

    • +
+
+
+

:raises (ElementError): if forbidden overlaps occur.

+
+ +
+
+integrate(sandbox: Sandbox) → None
+

Integrate currently staged filesystem against this artifact.

+
+
Parameters:
+

sandbox – The build sandbox

+
+
+

This modifies the sysroot staged inside the sandbox so that +the sysroot is integrated. Only an integrated sandbox +may be trusted for running the software therein, as the integration +commands will create and update important system cache files +required for running the installed software (such as the ld.so.cache).

+
+ +
+
+stage_sources(sandbox: Sandbox, directory: str) → None
+

Stage this element’s sources to a directory in the sandbox

+
+
Parameters:
+
    +

  • sandbox – The build sandbox

    • +

  • directory – An absolute path within the sandbox to stage the sources at

    • +
+
+
+
+ +
+
+get_public_data(domain: str) → MappingNode[Node]
+

Fetch public data on this element

+
+
Parameters:
+

domain – A public domain name to fetch data for

+
+
+

Returns:

+
+

Note

+

This can only be called the abstract methods which are +called as a part of the build phase +and never before.

+
+
+ +
+
+set_public_data(domain: str, data: MappingNode[Node]) → None
+

Set public data on this element

+
+
Parameters:
+
    +

  • domain – A public domain name to fetch data for

    • +

  • data – The public data dictionary for the given domain

    • +
+
+
+

This allows an element to dynamically mutate public data of +elements or add new domains as the result of success completion +of the Element.assemble() +method.

+
+ +
+
+get_environment() → Dict[str, str]
+

Fetch the environment suitable for running in the sandbox

+
+
Returns:
+

A dictionary of string key/values suitable for passing +to Sandbox.run()

+
+
+
+ +
+
+get_variable(varname: str) → str | None
+

Fetch the value of a variable resolved for this element.

+
+
Parameters:
+

varname – The name of the variable to fetch

+
+
Returns:
+

The resolved value for varname, or None if no +variable was declared with the given name.

+
+
+
+ +
+
+run_cleanup_commands(sandbox: Sandbox) → None
+

Run commands to cleanup the build directory.

+
+
Parameters:
+

sandbox – The build sandbox

+
+
+

This may be called at the end of a command batch in +Element.assemble() +to avoid the costs of capturing the build directory after a successful +build.

+

This will have no effect if the build tree is required after the build.

+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.exceptions.html b/buildstream.exceptions.html new file mode 100644 index 000000000..1f6ac2e51 --- /dev/null +++ b/buildstream.exceptions.html @@ -0,0 +1,508 @@ + + + + + + + + + Exceptions - API for Error Handling — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Exceptions - API for Error Handling

+

This module contains some Enums used in Error Handling which are useful in +testing external plugins.

+
+
+
+class ErrorDomain(value, names=None, *values, module=None, qualname=None, type=None, start=1, boundary=None)
+

Bases: Enum

+

Describes what the error is related to.

+
+
+PLUGIN = 1
+
+ +
+
+LOAD = 2
+
+ +
+
+IMPL = 3
+
+ +
+
+PLATFORM = 4
+
+ +
+
+SANDBOX = 5
+
+ +
+
+ARTIFACT = 6
+
+ +
+
+PIPELINE = 7
+
+ +
+
+UTIL = 8
+
+ +
+
+SOURCE = 9
+
+ +
+
+ELEMENT = 10
+
+ +
+
+APP = 11
+
+ +
+
+STREAM = 12
+
+ +
+
+VIRTUAL_FS = 13
+
+ +
+
+CAS = 14
+
+ +
+
+PROG_NOT_FOUND = 15
+
+ +
+
+REMOTE = 16
+
+ +
+
+PROFILE = 17
+
+ +
+ +
+
+class LoadErrorReason(value, names=None, *values, module=None, qualname=None, type=None, start=1, boundary=None)
+

Bases: Enum

+

Describes the reason why a LoadError was raised.

+
+
+MISSING_FILE = 1
+

A file was not found.

+
+ +
+
+INVALID_YAML = 2
+

The parsed data was not valid YAML.

+
+ +
+
+INVALID_DATA = 3
+

Data was malformed, a value was not of the expected type, etc

+
+ +
+
+ILLEGAL_COMPOSITE = 4
+

An error occurred during YAML dictionary composition.

+

This can happen by overriding a value with a new differently typed +value, or by overwriting some named value when that was not allowed.

+
+ +
+
+CIRCULAR_DEPENDENCY = 5
+

A circular dependency chain was detected

+
+ +
+
+UNRESOLVED_VARIABLE = 6
+

A variable could not be resolved. This can happen if your project +has cyclic dependencies in variable declarations, or, when substituting +a string which refers to an undefined variable.

+
+ +
+
+UNSUPPORTED_PROJECT = 7
+

The project requires an incompatible BuildStream version

+
+ +
+
+UNSUPPORTED_PLUGIN = 8
+

Project requires a newer version of a plugin than the one which was +loaded

+
+ +
+
+EXPRESSION_FAILED = 9
+

A conditional expression failed to resolve

+
+ +
+
+USER_ASSERTION = 10
+

An assertion was intentionally encoded into project YAML

+
+ +
+
+TRAILING_LIST_DIRECTIVE = 11
+

A list composition directive did not apply to any underlying list

+
+ +
+
+CONFLICTING_JUNCTION = 12
+

Conflicting junctions in subprojects

+
+ +
+
+INVALID_JUNCTION = 13
+

Failure to load a project from a specified junction

+
+ +
+
+SUBPROJECT_INCONSISTENT = 15
+

Subproject has no ref

+
+ +
+
+INVALID_SYMBOL_NAME = 16
+

An invalid symbol name was encountered

+
+ +
+
+MISSING_PROJECT_CONF = 17
+

A project.conf file was missing

+
+ +
+
+LOADING_DIRECTORY = 18
+

Try to load a directory not a yaml file

+
+ +
+
+PROJ_PATH_INVALID = 19
+

A project path leads outside of the project directory

+
+ +
+
+PROJ_PATH_INVALID_KIND = 20
+

A project path points to a file of the not right kind (e.g. a +socket)

+
+ +
+
+RECURSIVE_INCLUDE = 21
+

A recursive include has been encountered

+
+ +
+
+CIRCULAR_REFERENCE_VARIABLE = 22
+

A circular variable reference was detected

+
+ +
+
+PROTECTED_VARIABLE_REDEFINED = 23
+

An attempt was made to set the value of a protected variable

+
+ +
+
+INVALID_DEPENDENCY_CONFIG = 24
+

An attempt was made to specify dependency configuration on an element +which does not support custom dependency configuration

+
+ +
+ +

A link element declared dependencies

+
+ +
+
+CIRCULAR_REFERENCE = 26
+

A circular element reference was detected

+
+ +
+
+BAD_ELEMENT_SUFFIX = 27
+

This warning will be produced when an element whose name does not end in .bst +is referenced either on the command line or by another element

+
+ +
+
+BAD_CHARACTERS_IN_NAME = 28
+

This warning will be produced when a filename for a target contains invalid +characters in its name.

+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.html b/buildstream.html new file mode 100644 index 000000000..09091522d --- /dev/null +++ b/buildstream.html @@ -0,0 +1,615 @@ + + + + + + + + + buildstream package — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

buildstream package

+
+

Subpackages

+
+ +
+
+
+

Submodules

+
+ +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.node.html b/buildstream.node.html new file mode 100644 index 000000000..9ba132b45 --- /dev/null +++ b/buildstream.node.html @@ -0,0 +1,1097 @@ + + + + + + + + + Node - Parsed YAML configuration — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Node - Parsed YAML configuration

+

This module contains the building blocks for handling YAML configuration.

+

Everything that is loaded from YAML is encapsulated in such nodes, which +provide helper methods to validate configuration on access.

+

Using node methods when reading configuration will ensure that errors +are always coherently notified to the user.

+
+

Note

+

Plugins are not expected to handle exceptions thrown by node +methods for the above reason; They are private. There should +always be a way to acquire information without resorting to +exception handling.

+
+
+

Node types

+

The most important classes defined here are:

+
    +

  • MappingNode: represents a YAML Mapping (dictionary)

    • +

  • ScalarNode: represents a YAML Scalar (string, boolean, integer)

    • +

  • SequenceNode: represents a YAML Sequence (list)

    • +
+
+
+

Class Reference

+
+
+
+
+class MappingNode
+

Bases: Node

+

This class represents a Mapping (dict) in a YAML document.

+

It behaves mostly like a dict, but doesn’t allow untyped value access +(Nothing of the form my_dict[my_value].

+

It also doesn’t allow anything else than str as keys, to align with YAML.

+

You can however use common dict operations in it:

+
# Assign a new value to a key
+my_mapping[key] = my_value
+
+# Delete an entry
+del my_mapping[key]
+
+
+

When assigning a key/value pair, the key must be a string, +and the value can be any of:

+
    +

  • a Node, in which case the node is just assigned like normally

    • +

  • a list, dict, int, str, bool or None. +In which case, the value will be converted to a Node for you.

    • +
+

Therefore, all values in a MappingNode will be Node.

+
+

Note

+

You should never create an instance directly. Use Node.from_dict() +instead, which will ensure your node is correctly formatted.

+
+
+
+clone()
+

Clone the node and return the copy.

+
+
Returns:
+

a clone of the current node

+
+
Return type:
+

Node

+
+
+
+ +
+
+get_bool(key, default=sentinel)
+

Get the value of the node for key as a boolean.

+

This is equivalent to: mapping.get_scalar(my_key, my_default).as_bool().

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • default (bool) – default value to return if key is not in the mapping

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value at key is not a + ScalarNode or isn’t a + valid boolean

+
+
Returns:
+

the value at key or the default

+
+
Return type:
+

bool

+
+
+
+ +
+
+get_enum(key, constraint, default=<object object>)
+

Get the value of the node as an enum member from constraint

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • constraint (buildstream.types.FastEnum or Enum) – an enum from which to extract the value +for the current node.

    • +

  • default (object) – default value to return if key is not in the mapping

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value is not is not found or not part of the + provided enum.

+
+
Returns:
+

+
the value contained in the node, as a member of

constraint

+
+
+

+
+
Return type:
+

buildstream.types.Enum or Enum

+
+
+
+ +
+
+get_int(key, default=sentinel)
+

Get the value of the node for key as an integer.

+

This is equivalent to: mapping.get_scalar(my_key, my_default).as_int().

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • default (int, None) – default value to return if key is not in the mapping

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value at key is not a + ScalarNode or isn’t a + valid integer

+
+
Returns:
+

the value at key or the default

+
+
Return type:
+

int or None

+
+
+
+ +
+
+get_mapping(key, default=sentinel)
+

Get the value of the node for key as a MappingNode.

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • default (dict) – default value to return if key is not in the mapping. It will be converted +to a MappingNode before being returned

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value at key is not a + MappingNode

+
+
Returns:
+

the value at key or the default

+
+
Return type:
+

MappingNode

+
+
+
+ +
+
+get_node(key, allowed_types=None, allow_none=False)
+

Get the value of the node for key as a Node.

+

This is useful if you have configuration that can be either a ScalarNode or +a MappingNode for example.

+

This method will validate that the value is indeed exactly one of those types (not a subclass) +and raise an exception accordingly.

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • allowed_types (list) – list of valid subtypes of Node that are valid return values. +If this is None, no checks are done on the return value.

    • +

  • allow_none (bool) – whether to allow the return value to be None or not

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value at key is not one + of the expected types or if it doesn’t + exist.

+
+
Returns:
+

the value at key or None

+
+
Return type:
+

Node

+
+
+
+ +
+
+get_scalar(key, default=sentinel)
+

Get the value of the node for key as a ScalarNode.

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • default (str, int, bool, None) – default value to return if key is not in the mapping. +It will be converted to a ScalarNode before being +returned.

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value at key is not a + MappingNode

+
+
Returns:
+

the value at key or the default

+
+
Return type:
+

ScalarNode

+
+
+
+ +
+
+get_sequence(key, default=sentinel)
+

Get the value of the node for key as a SequenceNode.

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • default (list) – default value to return if key is not in the mapping. It will be converted +to a SequenceNode before being returned

    • +

  • allowed_types (list) – list of valid subtypes of Node that are valid for nodes in the sequence.

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value at key is not a + SequenceNode

+
+
Returns:
+

the value at key or the default

+
+
Return type:
+

SequenceNode

+
+
+
+ +
+
+get_str(key, default=sentinel)
+

Get the value of the node for key as an string.

+

This is equivalent to: mapping.get_scalar(my_key, my_default).as_str().

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • default (str) – default value to return if key is not in the mapping

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value at key is not a + ScalarNode or isn’t a + valid str

+
+
Returns:
+

the value at key or the default

+
+
Return type:
+

str

+
+
+
+ +
+
+get_str_list(key, default=sentinel)
+

Get the value of the node for key as a list of strings.

+

This is equivalent to: mapping.get_sequence(my_key, my_default).as_str_list().

+
+
Parameters:
+
    +

  • key (str) – key for which to get the value

    • +

  • default (str) – default value to return if key is not in the mapping

    • +
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value at key is not a + SequenceNode or if any + of its internal values is not a ScalarNode.

+
+
Returns:
+

the value at key or the default

+
+
Return type:
+

list

+
+
+
+ +
+
+items()
+

Get a new view of the mapping items ((key, value) pairs).

+

This is equivalent to running my_dict.item() on a dict.

+
+
Returns:
+

a view on the underlying dictionary

+
+
Return type:
+

dict_items

+
+
+
+ +
+
+keys()
+

Get the list of all keys in the mapping.

+

This is equivalent to running my_dict.keys() on a dict.

+
+
Returns:
+

a list of all keys in the mapping

+
+
Return type:
+

list

+
+
+
+ +
+
+safe_del(key)
+

Remove the entry at key in the dictionary if it exists.

+

This method is a safe equivalent to del mapping[key], that doesn’t +throw anything if the key doesn’t exist.

+
+
Parameters:
+

key (str) – key to remove from the mapping

+
+
+
+ +
+
+strip_node_info()
+

Remove all the node information (provenance) and return the underlying data as plain python objects

+
+
Returns:
+

the underlying data that was held in the node structure.

+
+
Return type:
+

(list, dict, str, None)

+
+
+
+ +
+
+validate_keys(valid_keys)
+

Validate that the node doesn’t contain extra keys

+

This validates the node so as to ensure the user has not specified +any keys which are unrecognized by BuildStream (usually this +means a typo which would otherwise not trigger an error).

+
+
Parameters:
+

valid_keys (list) – A list of valid keys for the specified node

+
+
Raises:
+

buildstream._exceptions.LoadError – In the case that the specified node contained + one or more invalid keys

+
+
+
+ +
+
+values()
+

Get the values in the mapping.

+

This is equivalent to running my_dict.values() on a dict.

+
+
Returns:
+

a list of all values in the mapping

+
+
Return type:
+

dict_values

+
+
+
+ +
+ +
+
+class Node
+

Bases: object

+

This is the base class for YAML document nodes.

+

YAML Nodes contain information to describe the provenance of the YAML +which resulted in the Node, allowing mapping back from a Node to the place +in the file it came from.

+
+

Note

+

You should never need to create a Node manually. +If you do, you can create Node from dictionaries with +Node.from_dict(). +If something else is needed, please open an issue.

+
+
+
+clone()
+

Clone the node and return the copy.

+
+
Returns:
+

a clone of the current node

+
+
Return type:
+

Node

+
+
+
+ +
+
+classmethod from_dict(value)
+

Create a new node from the given dictionary.

+

This is a recursive operation, and will transform every value in the +dictionary to a Node instance

+

Valid values for keys are str +Valid values for values are list, dict, str, int, bool or None. +list and dict can also only contain such types.

+
+
Parameters:
+

value (dict) – dictionary from which to create a node.

+
+
Raises:
+

TypeError – when the value cannot be converted to a Node

+
+
Returns:
+

a new mapping containing the value

+
+
Return type:
+

MappingNode

+
+
+
+ +
+
+get_provenance()
+

A convenience accessor to obtain the node’s ProvenanceInformation

+

The provenance information allows you to inform the user of where +a node came. Transforming the information to a string will show the file, line and column +in the file where the node is.

+

An example usage would be:

+
# With `config` being your node
+max_jobs_node = config.get_node('max-jobs')
+max_jobs = max_jobs_node.as_int()
+
+if max_jobs < 1:  # We can't get a negative number of jobs
+    raise LoadError("Error at {}: Max jobs needs to be >= 1".format(
+        max_jobs_node.get_provenance()
+    )
+
+# Will print something like:
+# element.bst [line 4, col 7]: Max jobs needs to be >= 1
+
+
+
+
Returns:
+

the provenance information for the node.

+
+
Return type:
+

ProvenanceInformation

+
+
+
+ +
+
+strip_node_info()
+

Remove all the node information (provenance) and return the underlying data as plain python objects

+
+
Returns:
+

the underlying data that was held in the node structure.

+
+
Return type:
+

(list, dict, str, None)

+
+
+
+ +
+ +
+
+class ProvenanceInformation
+

Bases: object

+

Represents the location of a YAML node in a file.

+

This can effectively be used as a pretty print to display those information in +errors consistently.

+

You can retrieve this information for a Node with +Node.get_provenance()

+
+ +
+
+class ScalarNode
+

Bases: Node

+

This class represents a Scalar (int, str, bool, None) in a YAML document.

+
+

Note

+

If you need to store another type of scalars, please open an issue +on the project.

+
+
+

Note

+

You should never have to create a ScalarNode directly

+
+
+
+as_bool()
+

Get the value of the node as a boolean.

+
+

Note

+

BuildStream treats the values ‘True’ and ‘true’ as True, +and the values ‘False’ and ‘false’ as False. Any other +string values (such as the valid YAML ‘TRUE’ or ‘FALSE’ +will be considered as an error)

+
+
+
Raises:
+

buildstream._exceptions.LoadError – if the value cannot be coerced to + a bool correctly.

+
+
Returns:
+

the value contained in the node, as a boolean

+
+
Return type:
+

bool

+
+
+
+ +
+
+as_enum(constraint)
+

Get the value of the node as an enum member from constraint

+

The constraint must be a buildstream.types.FastEnum or a plain python Enum.

+

For example you could do:

+
from buildstream.types import FastEnum
+
+class SupportedCompressions(FastEnum):
+  NONE = "none"
+  GZIP = "gzip"
+  XZ = "xz"
+
+
+x = config.get_scalar('compress').as_enum(SupportedCompressions)
+
+if x == SupportedCompressions.GZIP:
+    print("Using GZIP")
+
+
+
+
Parameters:
+

constraint (buildstream.types.FastEnum or Enum) – an enum from which to extract the value +for the current node.

+
+
Returns:
+

the value contained in the node, as a member of constraint

+
+
Return type:
+

FastEnum or Enum

+
+
+
+ +
+
+as_int()
+

Get the value of the node as an integer.

+
+
Raises:
+

buildstream._exceptions.LoadError – if the value cannot be coerced to + an integer correctly.

+
+
Returns:
+

the value contained in the node, as a integer

+
+
Return type:
+

int

+
+
+
+ +
+
+as_str()
+

Get the value of the node as a string.

+
+
Returns:
+

+
the value contained in the node, as a string, or None if the content

is None.

+
+
+

+
+
Return type:
+

str

+
+
+
+ +
+
+clone()
+

Clone the node and return the copy.

+
+
Returns:
+

a clone of the current node

+
+
Return type:
+

Node

+
+
+
+ +
+
+is_none()
+

Determine whether the current scalar is None.

+
+
Returns:
+

True if the value of the scalar is None, else False

+
+
Return type:
+

bool

+
+
+
+ +
+
+strip_node_info()
+

Remove all the node information (provenance) and return the underlying data as plain python objects

+
+
Returns:
+

the underlying data that was held in the node structure.

+
+
Return type:
+

(list, dict, str, None)

+
+
+
+ +
+ +
+
+class SequenceNode
+

Bases: Node

+

This class represents a Sequence (list) in a YAML document.

+

It behaves mostly like a list, but doesn’t allow untyped value access +(Nothing of the form my_list[my_value]).

+

You can however perform common list operations on it:

+
# Assign a value
+my_sequence[key] = value
+
+# Get the length
+len(my_sequence)
+
+# Reverse it
+reversed(my_sequence)
+
+# And iter over it
+for value in my_sequence:
+    print(value)
+
+
+

All values in a SequenceNode will be Node.

+
+
+append(value)
+

Append the given object to the sequence.

+
+
Parameters:
+

value (object) –

the value to append to the list. This can either be:

+
    +

  • a Node

    • +

  • a int, bool, str, None, +dict or list. In which case, this will be +converted into a Node beforehand

    • +
+

+
+
Raises:
+

TypeError – when the value cannot be converted to a Node

+
+
+
+ +
+
+as_str_list()
+

Get the values of the sequence as a list of strings.

+
+
Raises:
+

buildstream._exceptions.LoadError – if the sequence contains more than + ScalarNode

+
+
Returns:
+

the content of the sequence as a list of strings

+
+
Return type:
+

list

+
+
+
+ +
+
+clone()
+

Clone the node and return the copy.

+
+
Returns:
+

a clone of the current node

+
+
Return type:
+

Node

+
+
+
+ +
+
+mapping_at(index)
+

Retrieve the entry at index as a MappingNode.

+
+
Parameters:
+

index (int) – index for which to get the value

+
+
Raises:
+
    +

  • buildstream._exceptions.LoadError – if the value at key is not a + MappingNode

    • +

  • IndexError – if no value exists at this index

    • +
+
+
Returns:
+

the value at index

+
+
Return type:
+

MappingNode

+
+
+
+ +
+
+node_at(index, allowed_types=None)
+

Retrieve the entry at index as a Node.

+

This is useful if you have configuration that can be either a ScalarNode or +a MappingNode for example.

+

This method will validate that the value is indeed exactly one of those types (not a subclass) +and raise an exception accordingly.

+
+
Parameters:
+
    +

  • index (int) – index for which to get the value

    • +

  • allowed_types (list) – list of valid subtypes of Node that are valid return values. +If this is None, no checks are done on the return value.

    • +
+
+
Raises:
+
    +

  • buildstream._exceptions.LoadError – if the value at index is not of one of the + expected types

    • +

  • IndexError – if no value exists at this index

    • +
+
+
Returns:
+

the value at index

+
+
Return type:
+

Node

+
+
+
+ +
+
+scalar_at(index)
+

Retrieve the entry at index as a ScalarNode.

+
+
Parameters:
+

index (int) – index for which to get the value

+
+
Raises:
+
    +

  • buildstream._exceptions.LoadError – if the value at key is not a + ScalarNode

    • +

  • IndexError – if no value exists at this index

    • +
+
+
Returns:
+

the value at index

+
+
Return type:
+

ScalarNode

+
+
+
+ +
+
+sequence_at(index)
+

Retrieve the entry at index as a SequenceNode.

+
+
Parameters:
+

index (int) – index for which to get the value

+
+
Raises:
+
    +

  • buildstream._exceptions.LoadError – if the value at key is not a + SequenceNode

    • +

  • IndexError – if no value exists at this index

    • +
+
+
Returns:
+

the value at index

+
+
Return type:
+

SequenceNode

+
+
+
+ +
+
+strip_node_info()
+

Remove all the node information (provenance) and return the underlying data as plain python objects

+
+
Returns:
+

the underlying data that was held in the node structure.

+
+
Return type:
+

(list, dict, str, None)

+
+
+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.plugin.html b/buildstream.plugin.html new file mode 100644 index 000000000..057b3c20c --- /dev/null +++ b/buildstream.plugin.html @@ -0,0 +1,727 @@ + + + + + + + + + Plugin - Base plugin class — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Plugin - Base plugin class

+

BuildStream supports third party plugins to define additional kinds of +Elements and Sources.

+

The common API is documented here, along with some information on how +external plugin packages are structured.

+
+

Abstract Methods

+

For both Elements and Sources, +it is mandatory to implement the following abstract methods:

+
    +

  • Plugin.configure()

    +

    Loads the user provided configuration YAML for the given source or element

    +
    • +

  • Plugin.preflight()

    +

    Early preflight checks allow plugins to bail out early with an error +in the case that it can predict that failure is inevitable.

    +
    • +

  • Plugin.get_unique_key()

    +

    Once all configuration has been loaded and preflight checks have passed, +this method is used to inform the core of a plugin’s unique configuration.

    +
    • +
+
+
+

Configurable Warnings

+

Warnings raised through calling Plugin.warn() can provide an optional +parameter warning_token, this will raise a PluginError if the warning is configured as fatal within +the project configuration.

+

Configurable warnings will be prefixed with Plugin.get_kind() +within buildstream and must be prefixed as such in project configurations. For more detail on project configuration +see Configurable Warnings.

+

It is important to document these warnings in your plugin documentation to allow users to make full use of them +while configuring their projects.

+

Example

+

If the git plugin uses the warning "inconsistent-submodule" +then it could be referenced in project configuration as "git:inconsistent-submodule".

+
+
+

Plugin Structure

+

A plugin should consist of a setuptools package that +advertises contained plugins using entry points.

+

A plugin entry point must be a module that extends a class in the +Plugin API reference to be discovered by BuildStream. A YAML file +defining plugin default settings with the same name as the module can +also be defined in the same directory as the plugin module.

+
+

Note

+

BuildStream does not support function/class entry points.

+
+

A sample plugin could be structured as such:

+
.
+├── elements
+│   ├── autotools.py
+│   ├── autotools.yaml
+│   └── __init__.py
+├── MANIFEST.in
+└── setup.py
+
+
+

The setuptools configuration should then contain at least:

+

setup.py:

+
from setuptools import setup, find_packages
+
+setup(name='BuildStream Autotools',
+      version="0.1",
+      description="A better autotools element for BuildStream",
+      packages=find_packages(),
+      include_package_data=True,
+      entry_points={
+          'buildstream.plugins.elements': [
+              'autotools = elements.autotools'
+          ]
+      })
+
+
+

MANIFEST.in:

+
global-include *.yaml
+
+
+
+
+

Class Reference

+
+
+
+
+class Plugin
+

Bases: object

+

Base Plugin class.

+

Some common features to both Sources and Elements are found +in this class.

+
+

Note

+

Derivation of plugins is not supported. Plugins may only +derive from the base Source and +Element types, and any convenience +subclasses (like BuildElement) +which are included in the buildstream namespace.

+
+
+
+BST_MIN_VERSION: str | None = None
+

The minimum required version of BuildStream required by this plugin.

+

The version must be expressed as the string “<major>.<minor>”, where the +major version number is the API version and the minor version number is +the revision of the same BuildStream API where new symbols might have been +added to the API.

+

Example:

+

The following statement means that this plugin works with BuildStream 2.X, +only if X >= 2:

+
class Foo(Source):
+
+    # Our plugin requires 2.2
+    BST_MIN_VERSION = "2.2"
+
+
+
+

Note

+

This version works exactly the same was as the min-version +which must be specified in the project.conf file.

+
+
+ +
+
+BST_PLUGIN_DEPRECATED = False
+

True if this element plugin has been deprecated.

+

If this is set to true, BuildStream will emit a deprecation warning +in any place where this plugin is used.

+

The deprecation warnings can be suppressed when defining the +plugin origins in your project configuration

+
+ +
+
+BST_PLUGIN_DEPRECATION_MESSAGE = None
+

An additional message to report when a plugin is deprecated

+

This can be used to refer the user to a suitable replacement or +alternative approach when the plugin is deprecated.

+
+ +
+
+name
+

The plugin name

+

For elements, this is the project relative bst filename, +for sources this is the owning element’s name with a suffix +indicating its index on the owning element.

+

For sources this is for display purposes only.

+
+ +
+
+configure(node: MappingNode) → None
+

Configure the Plugin from loaded configuration data

+
+
Parameters:
+

node – The loaded configuration dictionary

+
+
Raises:
+
    +

  • .SourceError – If it’s a Source implementation

    • +

  • .ElementError – If it’s an Element implementation

    • +
+
+
+

Plugin implementors should implement this method to read configuration +data and store it.

+

The MappingNode.validate_keys() method +should be used to ensure that the user has not specified keys in node which are unsupported +by the plugin.

+
+ +
+
+preflight() → None
+

Preflight Check

+
+
Raises:
+
    +

  • .SourceError – If it’s a Source implementation

    • +

  • .ElementError – If it’s an Element implementation

    • +
+
+
+

This method is run after Plugin.configure() +and after the pipeline is fully constructed.

+

Implementors should simply raise SourceError or ElementError +with an informative message in the case that the host environment is +unsuitable for operation.

+

Plugins which require host tools (only sources usually) should obtain +them with utils.get_host_tool() which +will raise an error automatically informing the user that a host tool is needed.

+
+ +
+
+get_unique_key() → None | int | str | List[Any] | Dict[str, Any]
+

Return something which uniquely identifies the plugin input

+
+
Returns:
+

A string, list or dictionary which uniquely identifies the input

+
+
+

This is used to construct unique cache keys for elements +and sources, sources should return something which uniquely identifies the payload, +such as an sha256 sum of a tarball content.

+

Elements and Sources should implement this by collecting any configurations +which could possibly affect the output and return a dictionary of these settings.

+

For Sources, this is guaranteed to only be called if +Source.is_resolved() has returned +True which is to say that the Source is expected to have an exact +source ref indicating exactly what source is going to be staged.

+
+

Note

+

If your plugin is concerned with API stability, then future extensions of your +plugin YAML configuration which affect the unique key returned here should be added +to this key with care.

+

A good rule of thumb is to only compute the new value in the returned key if +the value of the newly added YAML key is not equal to it’s default value.

+
+
+ +
+
+get_kind() → str
+

Fetches the kind of this plugin

+
+
Returns:
+

The kind of this plugin

+
+
+
+ +
+
+node_get_project_path(node, *, check_is_file=False, check_is_dir=False) → str
+

Fetches a project path from a dictionary node and validates it

+

Paths are asserted to never lead to a directory outside of the +project directory. In addition, paths can not point to symbolic +links, fifos, sockets and block/character devices.

+

The check_is_file and check_is_dir parameters can be used to +perform additional validations on the path. Note that an +exception will always be raised if both parameters are set to +True.

+
+
Parameters:
+
    +

  • node (ScalarNode) – A Node loaded from YAML containing the path to validate

    • +

  • check_is_file (bool) – If True an error will also be raised +if path does not point to a regular file. +Defaults to False

    • +

  • check_is_dir (bool) – If True an error will also be raised +if path does not point to a directory. +Defaults to False

    • +
+
+
Returns:
+

The project path

+
+
Return type:
+

(str)

+
+
Raises:
+

.LoadError – In the case that the project path is not + valid or does not exist

+
+
+

Example:

+
path = self.node_get_project_path(node, 'path')
+
+
+
+ +
+
+debug(brief: str, *, detail: str | None = None) → None
+

Print a debugging message

+
+
Parameters:
+
    +

  • brief – The brief message

    • +

  • detail – An optional detailed message, can be multiline output

    • +
+
+
+
+ +
+
+status(brief: str, *, detail: str | None = None) → None
+

Print a status message

+
+
Parameters:
+
    +

  • brief – The brief message

    • +

  • detail – An optional detailed message, can be multiline output

    • +
+
+
+

Note: Status messages tell about what a plugin is currently doing

+
+ +
+
+info(brief: str, *, detail: str | None = None) → None
+

Print an informative message

+
+
Parameters:
+
    +

  • brief – The brief message

    • +

  • detail – An optional detailed message, can be multiline output

    • +
+
+
+
+
Note: Informative messages tell the user something they might want

to know, like if refreshing an element caused it to change. +The instance full name of the plugin will be generated with the +message, this being the name of the given element, as appose to +the class name of the underlying plugin __kind identifier.

+
+
+
+ +
+
+warn(brief: str, *, detail: str | None = None, warning_token: str | None = None) → None
+

Print a warning message, checks warning_token against project configuration

+
+
Parameters:
+
    +

  • brief – The brief message

    • +

  • detail – An optional detailed message, can be multiline output

    • +

  • warning_token – An optional configurable warning assosciated with this warning, +this will cause PluginError to be raised if this warning is configured as fatal.

    • +
+
+
+

:raises (PluginError): When warning_token is considered fatal by the project configuration

+
+ +
+
+log(brief: str, *, detail: str | None = None) → None
+

Log a message into the plugin’s log file

+

The message will not be shown in the master log at all (so it will not +be displayed to the user on the console).

+
+
Parameters:
+
    +

  • brief – The brief message

    • +

  • detail – An optional detailed message, can be multiline output

    • +
+
+
+
+ +
+
+timed_activity(activity_name: str, *, detail: str | None = None, silent_nested: bool = False) → Generator[None, None, None]
+

Context manager for performing timed activities in plugins

+
+
Parameters:
+
    +

  • activity_name – The name of the activity

    • +

  • detail – An optional detailed message, can be multiline output

    • +

  • silent_nested – If specified, nested messages will be silenced

    • +
+
+
+

This function lets you perform timed tasks in your plugin, +the core will take care of timing the duration of your +task and printing start / fail / success messages.

+

Example

+
# Activity will be logged and timed
+with self.timed_activity("Mirroring {}".format(self.url)):
+
+    # This will raise SourceError on its own
+    self.call(... command which takes time ...)
+
+
+
+ +
+
+blocking_activity(target: Callable[[...], T1], args: Sequence[Any], activity_name: str, *, detail: str | None = None, silent_nested: bool = False) → T1
+

Execute a blocking activity in the background.

+

This is to execute potentially blocking methods in the background, +in order to avoid starving the scheduler.

+

The function, its arguments and return value must all be pickleable, +as it will be run in another process. The function should not raise +an exception.

+

This should be used whenever there is a potential for a blocking +syscall to not return in a reasonable (<1s) amount of time. +For example, you would use this if you were doing a request to a +remote server, without a timeout.

+
+
Parameters:
+
    +

  • target – the function to execute in the background

    • +

  • args – positional arguments to pass to the method to call

    • +

  • activity_name – The name of the activity

    • +

  • detail – An optional detailed message, can be multiline output

    • +

  • silent_nested – If specified, nested messages will be silenced

    • +
+
+
Returns:
+

the return value from target.

+
+
+
+ +
+
+call(args: str | bytes | PathLike[str] | PathLike[bytes] | Sequence[str | bytes | PathLike[str] | PathLike[bytes]], fail: str | None = None, fail_temporarily: bool = False, cwd: str | bytes | PathLike[str] | PathLike[bytes] | None = None, env: Dict[str, str] | None = None, stdin: None | int | IO[Any] = None, stdout: None | int | IO[Any] = None, stderr: None | int | IO[Any] = None) → int
+

A wrapper for subprocess.call()

+
+
Parameters:
+
    +

  • args – A sequence of program arguments or else a string or path-like-object

    • +

  • fail – A message to display if the process returns a non zero exit code

    • +

  • fail_temporarily – Whether any exceptions should be raised as temporary.

    • +

  • cwd – Optionally specify the working directory for the command

    • +

  • env – Optionally specify some environment variables for the command

    • +

  • stdin – Optionally specify standard input for the command

    • +

  • stdout – Optionally specify standard output for the command

    • +

  • stderr – Optionally specify standard error for the command

    • +
+
+
Returns:
+

The process exit code.

+
+
+

:raises (PluginError): If a non-zero return code is received and fail is specified

+
+

Attention

+

If fail is not specified, then the return value of subprocess.call() +is returned even on error, and no exception is automatically raised.

+
+
+

Attention

+

If stderr and/or stdout are specified, then these streams are delivered +directly to the plugin instead of being delivered to the associated log file.

+
+

Example

+
# Call some host tool
+self.tool = utils.get_host_tool('toolname')
+self.call(
+    [self.tool, '--download-ponies', self.mirror_directory],
+    "Failed to download ponies from {}".format(
+        self.mirror_directory))
+
+
+
+ +
+
+check_output(args: str | bytes | PathLike[str] | PathLike[bytes] | Sequence[str | bytes | PathLike[str] | PathLike[bytes]], fail: str | None = None, fail_temporarily: bool = False, cwd: str | bytes | PathLike[str] | PathLike[bytes] | None = None, env: Dict[str, str] | None = None, stdin: None | int | IO[Any] = None, stderr: None | int | IO[Any] = None) → Tuple[int, str]
+

A wrapper for subprocess.check_output()

+
+
Parameters:
+
    +

  • args –

    A sequence of program arguments or else a string or path-like-object

    +

    • +

  • fail – A message to display if the process returns a non zero exit code

    • +

  • fail_temporarily – Whether any exceptions should be raised as temporary.

    • +

  • cwd – Optionally specify the working directory for the command

    • +

  • env – Optionally specify some environment variables for the command

    • +

  • stdin – Optionally specify standard input for the command

    • +

  • stderr – Optionally specify standard error for the command

    • +
+
+
Returns:
+

A 2-tuple of form (process exit code, process standard output)

+
+
+

:raises (PluginError): If a non-zero return code is received and fail is specified

+
+

Attention

+

If fail is not specified, then the return value of subprocess.call() +is returned even on error, and no exception is automatically raised.

+
+
+

Attention

+

If stderr and/or stdout are specified, then these streams are delivered +directly to the plugin instead of being delivered to the associated log file.

+
+

Example

+
# Get the tool at preflight time
+self.tool = utils.get_host_tool('toolname')
+
+# Call the tool, automatically raise an error
+_, output = self.check_output(
+    [self.tool, '--print-ponies'],
+    "Failed to print the ponies in {}".format(
+        self.mirror_directory),
+    cwd=self.mirror_directory)
+
+# Call the tool, inspect exit code
+exit_code, output = self.check_output(
+    [self.tool, 'get-ref', tracking],
+    cwd=self.mirror_directory)
+
+if exit_code == 128:
+    return
+elif exit_code != 0:
+    fmt = "{plugin}: Failed to get ref for tracking: {track}"
+    raise SourceError(
+        fmt.format(plugin=self, track=tracking)) from e
+
+
+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.sandbox.html b/buildstream.sandbox.html new file mode 100644 index 000000000..dce5bc536 --- /dev/null +++ b/buildstream.sandbox.html @@ -0,0 +1,150 @@ + + + + + + + + + buildstream.sandbox package — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+ +
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.sandbox.sandbox.html b/buildstream.sandbox.sandbox.html new file mode 100644 index 000000000..c3511649f --- /dev/null +++ b/buildstream.sandbox.sandbox.html @@ -0,0 +1,334 @@ + + + + + + + + + Sandbox - The build sandbox — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Sandbox - The build sandbox

+

Element plugins which want to interface with the sandbox +need only understand this interface, while it may be given a different +sandbox implementation, any sandbox implementation it is given will +conform to this interface.

+

See also: Sandboxing.

+
+
+
+exception SandboxCommandError(message, *, detail=None, collect=None, reason='command-failed')
+

Bases: SandboxError

+

Raised by Sandbox implementations when a command fails.

+
+
Parameters:
+
    +

  • message (str) – The error message to report to the user

    • +

  • detail (str) – The detailed error string

    • +

  • collect (str) – An optional directory containing partial install contents

    • +

  • reason (str) – An optional reason string (defaults to ‘command-failed’)

    • +
+
+
+
+ +
+
+class Sandbox
+

Bases: object

+

Sandbox programming interface for Element plugins.

+
+
+get_virtual_directory() → Directory
+

Fetches the sandbox root directory as a virtual Directory.

+

The root directory is where artifacts for the base +runtime environment should be staged.

+
+
Returns:
+

The sandbox root directory

+
+
+
+ +
+
+set_environment(environment: Dict[str, str]) → None
+

Sets the environment variables for the sandbox

+
+
Parameters:
+

environment – The environment variables to use in the sandbox

+
+
+
+ +
+
+set_work_directory(directory: str) → None
+

Sets the work directory for commands run in the sandbox

+
+
Parameters:
+

directory – An absolute path within the sandbox

+
+
+
+ +
+
+mark_directory(directory: str) → None
+

Marks a sandbox directory and ensures it will exist

+
+
Parameters:
+

directory – An absolute path within the sandbox to mark

+
+
+
+

Note

+

Any marked directories will be read-write in the sandboxed +environment, only the root directory is allowed to be readonly.

+
+
+ +
+
+run(command: List[str], *, root_read_only: bool = False, cwd: str | None = None, env: Dict[str, str] | None = None, label: str = None) → int | None
+

Run a command in the sandbox.

+

If this is called outside a batch context, the command is immediately +executed.

+

If this is called in a batch context, the command is added to the batch +for later execution. If the command fails, later commands will not be +executed. Command flags must match batch flags.

+
+
Parameters:
+
    +

  • command – The command to run in the sandboxed environment, as a list +of strings starting with the binary to run.

    • +

  • root_read_only – Whether the sandbox root should be readonly.

    • +

  • cwd – The sandbox relative working directory in which to run the command.

    • +

  • env – A dictionary of string key, value pairs to set as environment +variables inside the sandbox environment.

    • +

  • label – An optional label for the command, used for logging.

    • +
+
+
Returns:
+

The program exit code, or None if running in batch context.

+
+
+

:raises (ProgramNotFoundError): If a host tool which the given sandbox + implementation requires is not found.

+
+

Note

+

The optional cwd argument will default to the value set with +set_work_directory() and this +function must make sure the directory will be created if it does +not exist yet, even if a workspace is being used.

+
+
+ +
+
+batch(*, root_read_only: bool = False, label: str = None, collect: str = None) → Generator[None, None, None]
+

Context manager for command batching

+

This provides a batch context that defers execution of commands until +the end of the context. If a command fails, the batch will be aborted +and subsequent commands will not be executed.

+

Command batches may be nested. Execution will start only when the top +level batch context ends.

+
+
Parameters:
+
    +

  • root_read_only – Whether the sandbox root should be readonly.

    • +

  • label – An optional label for the batch group, used for logging.

    • +

  • collect – An optional directory containing partial install contents +on command failure.

    • +
+
+
+

:raises (SandboxCommandError): If a command fails.

+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.scriptelement.html b/buildstream.scriptelement.html new file mode 100644 index 000000000..28b082304 --- /dev/null +++ b/buildstream.scriptelement.html @@ -0,0 +1,435 @@ + + + + + + + + + ScriptElement - Abstract class for scripting elements — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

ScriptElement - Abstract class for scripting elements

+

The ScriptElement class is a convenience class one can derive for +implementing elements that stage elements and run command-lines on them.

+

Any derived classes must write their own configure() implementation, using +the public APIs exposed in this class.

+

Derived classes must also chain up to the parent method in their preflight() +implementations.

+
+
+
+class ScriptElement(context: Context, project: Project, load_element: LoadElement, plugin_conf: str | None, *, artifact_key: str = None)
+

Bases: Element

+
+
+BST_STRICT_REBUILD = True
+

Whether to rebuild this element in non strict mode if +any of the dependencies have changed.

+
+ +
+
+BST_FORBID_RDEPENDS = True
+

Whether to raise exceptions if an element has runtime dependencies.

+
+ +
+
+BST_FORBID_SOURCES = True
+

Whether to raise exceptions if an element has sources.

+
+ +
+
+set_work_dir(work_dir: str | None = None) → None
+

Sets the working dir

+

The working dir (a.k.a. cwd) is the directory which commands will be +called from.

+
+
Parameters:
+

work_dir – The working directory. If called without this argument +set, it’ll default to the value of the variable cwd.

+
+
+
+ +
+
+set_install_root(install_root: str | None = None) → None
+

Sets the install root

+

The install root is the directory which output will be collected from +once the commands have been run.

+
+
Parameters:
+

install_root – The install root. If called without this argument +set, it’ll default to the value of the variable install-root.

+
+
+
+ +
+
+set_root_read_only(root_read_only: bool) → None
+

Sets root read-only

+

When commands are run, if root_read_only is true, then the root of the +filesystem will be protected. This is strongly recommended whenever +possible.

+

If this variable is not set, the default permission is read-write.

+
+
Parameters:
+

root_read_only – Whether to mark the root filesystem as read-only.

+
+
+
+ +
+
+layout_add(element: Element, dependency_path: str, location: str) → None
+

Adds an element to the layout.

+

Layout is a way of defining how dependencies should be added to the +staging area for running commands.

+
+
Parameters:
+
    +

  • element (Element) – The element to stage.

    • +

  • dependency_path (str) – The element relative path to the dependency, usually obtained via +the dependency configuration

    • +

  • location (str) – The path inside the staging area for where to +stage this element. If it is not “/”, then integration +commands will not be run.

    • +
+
+
+

If this function is never called, then the default behavior is to just +stage the build dependencies of the element in question at the +sandbox root. Otherwise, the specified elements including their +runtime dependencies will be staged in their respective locations.

+
+

Note

+

The order of directories in the layout is not significant.

+

The paths in the layout will be sorted so that elements are staged in parent +directories before subdirectories.

+

The elements for each respective staging directory in the layout will be staged +in the predetermined deterministic staging order.

+
+
+ +
+
+add_commands(group_name: str, command_list: List[str]) → None
+

Adds a list of commands under the group-name.

+
+

Note

+

Command groups will be run in the order they were added.

+
+
+

Note

+

This does not perform substitutions automatically. They must +be performed beforehand (see +node_subst_list())

+
+
+
Parameters:
+
    +

  • group_name (str) – The name of the group of commands.

    • +

  • command_list (list) – The list of commands to be run.

    • +
+
+
+
+ +
+
+preflight()
+

Preflight Check

+
+
Raises:
+
    +

  • .SourceError – If it’s a Source implementation

    • +

  • .ElementError – If it’s an Element implementation

    • +
+
+
+

This method is run after Plugin.configure() +and after the pipeline is fully constructed.

+

Implementors should simply raise SourceError or ElementError +with an informative message in the case that the host environment is +unsuitable for operation.

+

Plugins which require host tools (only sources usually) should obtain +them with utils.get_host_tool() which +will raise an error automatically informing the user that a host tool is needed.

+
+ +
+
+get_unique_key()
+

Return something which uniquely identifies the plugin input

+
+
Returns:
+

A string, list or dictionary which uniquely identifies the input

+
+
+

This is used to construct unique cache keys for elements +and sources, sources should return something which uniquely identifies the payload, +such as an sha256 sum of a tarball content.

+

Elements and Sources should implement this by collecting any configurations +which could possibly affect the output and return a dictionary of these settings.

+

For Sources, this is guaranteed to only be called if +Source.is_resolved() has returned +True which is to say that the Source is expected to have an exact +source ref indicating exactly what source is going to be staged.

+
+

Note

+

If your plugin is concerned with API stability, then future extensions of your +plugin YAML configuration which affect the unique key returned here should be added +to this key with care.

+

A good rule of thumb is to only compute the new value in the returned key if +the value of the newly added YAML key is not equal to it’s default value.

+
+
+ +
+
+configure_sandbox(sandbox)
+

Configures the the sandbox for execution

+
+
Parameters:
+

sandbox – The build sandbox

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to configure the sandbox object +for execution.

+
+ +
+
+stage(sandbox)
+

Stage inputs into the sandbox directories

+
+
Parameters:
+

sandbox – The build sandbox

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to populate the sandbox +directory with data. This is done either by staging Source +objects, by staging the artifacts of the elements this element depends +on, or both.

+
+ +
+
+assemble(sandbox)
+

Assemble the output artifact

+
+
Parameters:
+

sandbox – The build sandbox

+
+
Returns:
+

An absolute path within the sandbox to collect the artifact from

+
+
+

:raises (ElementError): When the element raises an error

+

Elements must implement this method to create an output +artifact from its sources and dependencies.

+
+ +
+ +
+
+setup()
+
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.source.html b/buildstream.source.html new file mode 100644 index 000000000..b8d526704 --- /dev/null +++ b/buildstream.source.html @@ -0,0 +1,888 @@ + + + + + + + + + Source - Base source class — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Source - Base source class

+
+

Built-in functionality

+

The Source base class provides built in functionality that may be overridden +by individual plugins.

+
    +

  • Directory

    +

    The directory variable can be set for all sources of a type in project.conf +or per source within a element.

    +

    This sets the location within the build root that the content of the source +will be loaded in to. If the location does not exist, it will be created.

    +
    • +
+
+
+

Abstract Methods

+

For loading and configuration purposes, Sources must implement the +Plugin base class abstract methods.

+
+

Attention

+

In order to ensure that all configuration data is processed at +load time, it is important that all URLs have been processed during +Plugin.configure().

+

Source implementations must either call +Source.translate_url() or +Source.mark_download_url() +for every URL that has been specified in the configuration during +Plugin.configure()

+
+

Sources expose the following abstract methods. Unless explicitly mentioned, +these methods are mandatory to implement.

+ +
+
+

Working with the source ref

+

The SourceRef is used to determine the exact +version of data to be addressed by the source.

+

The various responsibilities involving the source reference are described here.

+
+

Loading and saving

+

The source reference is expected to be loaded at +Plugin.configure() and +and Source.load_ref() time +from the provided MappingNode.

+

The SourceRef should be loaded from a single key +in that node, the recommended name for that key is ref, but is ultimately up +to the implementor to decide.

+

When Source.set_ref() is called, +the source reference should be assigned to the same single key in the +provided MappingNode, this will be used to serialize changed +source references to YAML as a result of tracking.

+
+
+

Tracking new references

+

When the user tracks for new versions of the source, +then the new SourceRef should be returned from +the Source.track() implementation.

+
+
+

Managing internal state

+

Internally the source implementation is expected to keep track of its +SourceRef. The internal state should be +updated when Plugin.configure(), +Source.load_ref() or +Source.set_ref() is called.

+

The internal state should not be updated when +Source.track() is called.

+

The internal source ref must be returned on demand whenever +Source.get_ref() is called.

+
+
+

Generating the unique key

+

When Plugin.get_unique_key() +is called, the source’s SourceRef must be considered +as a part of that key.

+

The unique key will be used to generate the cache key of cache keys +of elements using this source, and so the unique key should be comprised of every +configuration which may effect how the source is staged, +as well as any configuration which uniquely identifies the source, which of course +includes the SourceRef.

+
+
+
+

Accessing previous sources

+

In the general case, all sources are fetched and tracked independently of one +another. In situations where a source needs to access previous source(s) in +order to perform its own track and/or fetch, following attributes can be set to +request access to previous sources:

+ +

The intended use of such plugins is to fetch external dependencies of other +sources, typically using some kind of package manager, such that all the +dependencies of the original source(s) are available at build time.

+

When implementing such a plugin, implementors should adhere to the following +guidelines:

+
    +

  • Implementations must be able to store the obtained artifacts in a +subdirectory.

    • +

  • Implementations must be able to deterministically generate a unique ref, such +that two refs are different if and only if they produce different outputs.

    • +

  • Implementations must not introduce host contamination.

    • +
+
+
+
+

SourceFetcher - Object for fetching individual URLs

+
+

Abstract Methods

+

SourceFetchers expose the following abstract methods. Unless explicitly +mentioned, these methods are mandatory to implement.

+
    +

  • SourceFetcher.fetch()

    +

    Fetches the URL associated with this SourceFetcher, optionally taking an +alias override.

    +
    • +
+
+
+

Class Reference

+
+
+
+
+exception SourceError(message: str, *, detail: str | None = None, reason: str | None = None, temporary: bool = False)
+

Bases: BstError

+

This exception should be raised by Source implementations +to report errors to the user.

+
+
Parameters:
+
    +

  • message – The breif error description to report to the user

    • +

  • detail – A possibly multiline, more detailed error message

    • +

  • reason – An optional machine readable reason string, used for test cases

    • +

  • temporary – An indicator to whether the error may occur if the operation was run again.

    • +
+
+
+
+ +
+
+class AliasSubstitution(_effective_alias: str, _mirror: buildstream.sourcemirror.SourceMirror | str)
+

Bases: object

+
+ +
+
+class SourceFetcher
+

Bases: object

+

This interface exists so that a source that downloads from multiple +places (e.g. a git source with submodules) has a consistent interface for +fetching and substituting aliases.

+
+

Attention

+

When implementing a SourceFetcher, remember to call +Source.mark_download_url() +for every URL found in the configuration data at +Plugin.configure() time.

+
+
+
+fetch(alias_override: AliasSubstitution | None = None, **kwargs) → None
+

Fetch remote sources and mirror them locally, ensuring at least +that the specific reference is cached locally.

+
+
Parameters:
+

alias_override – The alias to use instead of the default one +defined by the aliases field +in the project’s config.

+
+
Raises:
+

.SourceError –

+
+
+

Implementors should raise SourceError if the there is some +network error or if the source reference could not be matched.

+
+ +
+
+mark_download_url(url: str) → None
+

Identifies the URL that this SourceFetcher uses to download

+

This must be called during the fetcher’s initialization

+
+
Parameters:
+

url – The url used to download.

+
+
+
+

Note

+

While this must be called in a SourceFetcher initializer for the URL which +will be used by the fetcher, note that any URLs which are known and specified +in the Source configuration YAML must be marked with either +Source.mark_download_url() or +Source.translate_url() in +the Plugin.configure() implementation.

+
+
+ +
+ +
+
+class Source
+

Bases: Plugin

+

Base Source class.

+

All Sources derive from this class, this interface defines how +the core will be interacting with Sources.

+
+
+BST_REQUIRES_PREVIOUS_SOURCES_TRACK = False
+

Whether access to previous sources is required during track

+
+
When set to True:
    +

  • all sources listed before this source in the given element will be +fetched before this source is tracked

    • +

  • Source.track() will be called with an additional keyword argument +previous_sources_dir where previous sources will be staged

    • +

  • this source can not be the first source for an element

    • +
+
+
+
+ +
+
+BST_REQUIRES_PREVIOUS_SOURCES_FETCH = False
+

Whether access to previous sources is required during fetch

+
+
When set to True:
    +

  • all sources listed before this source in the given element will be +fetched before this source is fetched

    • +

  • Source.fetch() will be called with an additional keyword argument +previous_sources_dir where previous sources will be staged

    • +

  • this source can not be the first source for an element

    • +
+
+
+
+ +
+
+BST_REQUIRES_PREVIOUS_SOURCES_STAGE = False
+

Whether access to previous sources is required during cache

+
+
When set to True:
    +

  • All sources listed before current source in the given element will be +staged with the source when it’s cached.

    • +

  • This source can not be the first source for an element.

    • +
+
+
+
+ +
+
+BST_STAGE_VIRTUAL_DIRECTORY = False
+

Whether we can stage this source directly to a virtual directory

+

When set to True, Source.stage_directory() +and Source.init_workspace_directory() +will be called in place of Source.stage() and +Source.init_workspace() respectively.

+
+ +
+
+COMMON_CONFIG_KEYS = ['kind', 'directory']
+

Common source config keys

+

Source config keys that must not be accessed in configure(), and +should be checked for using node.validate_keys().

+
+ +
+
+load_ref(node: MappingNode) → None
+

Loads the SourceRef for this Source from the specified node.

+
+
Parameters:
+

node – The YAML node to load the ref from

+
+
+

Working with the source ref is discussed here.

+
+

Note

+

The SourceRef for the Source is expected to be read at +Plugin.configure() time, +this will only be used for loading refs from alternative locations +than in the element.bst file where the given Source object has +been declared.

+
+
+ +
+
+get_ref() → None | int | str | List[Any] | Dict[str, Any]
+

Fetch the SourceRef

+
+
Returns:
+

The internal SourceRef, or None

+
+
+

Working with the source ref is discussed here.

+
+ +
+
+set_ref(ref: None | int | str | List[Any] | Dict[str, Any], node: MappingNode) → None
+

Applies the internal ref, however it is represented

+
+
Parameters:
+
+
+
+

The implementor must update the node parameter to reflect the new ref, +and it should store the passed ref so that it will be returned in any +later calls to Source.get_ref().

+

The passed ref parameter is guaranteed to either be a value which has +been previously retrieved by the Source.get_ref() +method on the same plugin, or None.

+

Example:

+
# Implementation of Source.set_ref()
+#
+def set_ref(self, ref, node):
+
+    # Update internal state of the ref
+    self.ref = ref
+
+    # Update the passed node so that we will read the new ref
+    # next time this source plugin is configured with this node.
+    #
+    node["ref"] = self.ref
+
+
+

Working with the source ref is discussed here.

+
+ +
+
+track(*, previous_sources_dir: str = None) → None | int | str | List[Any] | Dict[str, Any]
+

Resolve a new ref from the plugin’s track option

+
+
Parameters:
+

previous_sources_dir (str) – directory where previous sources are staged. +Note that this keyword argument is available only when +BST_REQUIRES_PREVIOUS_SOURCES_TRACK +is set to True.

+
+
Returns:
+

A new SourceRef, or None

+
+
+

If the backend in question supports resolving references from +a symbolic tracking branch or tag, then this should be implemented +to perform this task on behalf of bst source track +commands.

+

This usually requires fetching new content from a remote origin +to see if a new ref has appeared for your branch or tag. If the +backend store allows one to query for a new ref from a symbolic +tracking data without downloading then that is desirable.

+

Working with the source ref is discussed here.

+
+ +
+
+fetch(*, previous_sources_dir: str = None) → None
+

Fetch remote sources and mirror them locally, ensuring at least +that the specific reference is cached locally.

+
+
Parameters:
+

previous_sources_dir (str) – directory where previous sources are staged. +Note that this keyword argument is available only when +BST_REQUIRES_PREVIOUS_SOURCES_FETCH +is set to True.

+
+
Raises:
+

.SourceError –

+
+
+

Implementors should raise SourceError if the there is some +network error or if the source reference could not be matched.

+
+ +
+
+stage(directory: str) → None
+

Stage the sources to a directory

+
+
Parameters:
+

directory – Path to stage the source

+
+
Raises:
+

.SourceError –

+
+
+

Implementors should assume that directory already exists +and stage already cached sources to the passed directory.

+

Implementors should raise SourceError when encountering +some system error.

+
+ +
+
+stage_directory(directory: Directory) → None
+

Stage the sources to a directory

+
+
Parameters:
+

directory – Directory object to stage the source into

+
+
Raises:
+

.SourceError –

+
+
+

Implementors should assume that directory represents an existing +directory root into which the source content can be populated.

+

Implementors should raise SourceError when encountering +some system error.

+
+

Note

+

This will be called instead of Source.stage() +in the case that BST_STAGE_VIRTUAL_DIRECTORY is set +for this plugin.

+
+
+ +
+
+init_workspace(directory: str) → None
+

Stage sources for use as a workspace.

+
+
Parameters:
+

directory – Path of the workspace to initialize.

+
+
Raises:
+

.SourceError –

+
+
+

Default implementation is to call +Source.stage().

+

Implementors overriding this method should assume that directory +already exists.

+

Implementors should raise SourceError when encountering +some system error.

+
+ +
+
+init_workspace_directory(directory: Directory) → None
+

Stage sources for use as a workspace.

+
+
Parameters:
+

directory – Directory object of the workspace to initialize.

+
+
Raises:
+

.SourceError –

+
+
+

Default implementation is to call +Source.stage_directory().

+

Implementors overriding this method should assume that directory +already exists.

+

Implementors should raise SourceError when encountering +some system error.

+
+

Note

+

This will be called instead of +Source.init_workspace() in the case that +BST_STAGE_VIRTUAL_DIRECTORY is set for this plugin.

+
+
+ +
+
+get_source_fetchers() → Iterable[SourceFetcher]
+

Get the objects that are used for fetching

+

If this source doesn’t download from multiple URLs, +returning None and falling back on the default behaviour +is recommended.

+
+
Returns:
+

The Source’s SourceFetchers, if any.

+
+
+
+

Note

+

Implementors can implement this as a generator.

+

The SourceFetcher.fetch() +method will be called on the returned fetchers one by one, +before consuming the next fetcher in the list.

+
+
+ +
+
+validate_cache() → None
+

Implement any validations once we know the sources are cached

+

This is guaranteed to be called only once for a given session +once the sources are known to be cached, before +Source.stage() or +Source.init_workspace() +is called.

+
+ +
+
+is_cached() → bool
+

Get whether the source has a local copy of its data.

+

This method is guaranteed to only be called whenever +Source.is_resolved() +returns True.

+

Returns: whether the source is cached locally or not.

+
+ +
+
+get_mirror_directory() → str
+

Fetches the directory where this source should store things

+
+
Returns:
+

The directory belonging to this source

+
+
+
+ +
+
+translate_url(url: str, *, alias_override: AliasSubstitution | None = None, primary: bool = True, suffix: str | None = None, extra_data: Dict[str, Any] | None = None) → str
+

Translates the given url which may be specified with an alias +into a fully qualified url.

+
+
Parameters:
+
    +

  • url – A URL, which may be using an alias

    • +

  • alias_override – Optionally, an URI to override the alias with.

    • +

  • primary – Whether this is the primary URL for the source.

    • +

  • suffix – an optional suffix to append to the URL (Since: 2.2)

    • +

  • extra_data – Additional data provided by SourceMirror (Since: 2.2)

    • +
+
+
Returns:
+

The fully qualified URL, with aliases resolved

+
+
+
+

Note

+

This must be called for every URL in the configuration during +Plugin.configure() if +Source.mark_download_url() +is not called.

+

The suffix argument may be used to translate URLs for which only the base portion of +the URL was previously marked with Source.mark_download_url() +at Plugin.configure() time.

+
+
+ +
+
+mark_download_url(url: str, *, primary: bool = True) → None
+

Identifies the URL that this Source uses to download

+
+
Parameters:
+
    +

  • url (str) – The URL used to download

    • +

  • primary (bool) – Whether this is the primary URL for the source

    • +
+
+
+
+

Note

+

This must be called for every URL in the configuration during +Plugin.configure() if +Source.translate_url() +is not called.

+
+
+ +
+
+get_project_directory() → str
+

Fetch the project base directory

+

This is useful for sources which need to load resources +stored somewhere inside the project.

+
+
Returns:
+

The project base directory

+
+
+
+ +
+
+tempdir() → Iterator[str]
+

Context manager for working in a temporary directory

+
+
Yields:
+

A path to a temporary directory

+
+
+

This should be used by source plugins directly instead of the tempfile +module. This one will automatically cleanup in case of termination by +catching the signal before os._exit(). It will also use the ‘mirror +directory’ as expected for a source.

+
+ +
+
+is_resolved() → bool
+

Get whether the source is resolved.

+

This has a default implementation that checks whether the source +has a ref or not. If it has a ref, it is assumed to be resolved.

+

Sources that never have a ref or have uncommon requirements can +override this method to specify when they should be considered +resolved

+

Returns: whether the source is fully resolved or not

+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.sourcemirror.html b/buildstream.sourcemirror.html new file mode 100644 index 000000000..781d04bc2 --- /dev/null +++ b/buildstream.sourcemirror.html @@ -0,0 +1,299 @@ + + + + + + + + + SourceMirror - Base source mirror class — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

SourceMirror - Base source mirror class

+

The SourceMirror plugin allows one to customize how +Source.translate_url() will +behave when looking up mirrors, allowing some additional flexibility in the +implementation of source mirrors.

+
+

Abstract Methods

+

For loading and configuration purposes, SourceMirrors may optionally implement +the Plugin base class Plugin.configure() method +in order to load any custom configuration in the config dictionary.

+

The remaining Plugin base class abstract methods are +not relevant to the SourceMirror plugin object and need not be implemented.

+

SourceMirrors expose the following abstract methods. Unless explicitly mentioned, +these methods are mandatory to implement.

+
    +

  • SourceMirror.translate_url()

    +

    Produce an appropriate URL for the given URL and alias.

    +
    • +
+
+
+

Class Reference

+
+
+
+
+exception SourceMirrorError(message: str, *, detail: str | None = None, reason: str | None = None, temporary: bool = False)
+

Bases: BstError

+

This exception should be raised by SourceMirror implementations +to report errors to the user.

+
+
Parameters:
+
    +

  • message – The breif error description to report to the user

    • +

  • detail – A possibly multiline, more detailed error message

    • +

  • reason – An optional machine readable reason string, used for test cases

    • +
+
+
+

Since: 2.2

+
+ +
+
+class SourceMirror
+

Bases: Plugin

+

Base SourceMirror class.

+

All SourceMirror plugins derive from this class, this interface defines how +the core will be interacting with SourceMirror plugins.

+

Since: 2.2

+
+
+BST_MIN_VERSION: str | None = '2.2'
+

The minimum required version of BuildStream required by this plugin.

+

The version must be expressed as the string “<major>.<minor>”, where the +major version number is the API version and the minor version number is +the revision of the same BuildStream API where new symbols might have been +added to the API.

+

Example:

+

The following statement means that this plugin works with BuildStream 2.X, +only if X >= 2:

+
class Foo(Source):
+
+    # Our plugin requires 2.2
+    BST_MIN_VERSION = "2.2"
+
+
+
+

Note

+

This version works exactly the same was as the min-version +which must be specified in the project.conf file.

+
+
+ +
+
+set_supported_aliases(aliases: List[str])
+

Set the aliases for which self can translate urls.

+
+
Parameters:
+

aliases – The list of aliases supported by this plugin

+
+
+
+ +
+
+translate_url(*, alias: str, alias_url: str, source_url: str, extra_data: Dict[str, Any] | None) → str
+

Produce an alternative url for url for the given alias.

+

This method implements the behavior of Source.translate_url().

+
+
Parameters:
+
    +

  • alias – The alias to translate for

    • +

  • alias_url – The default URL configured for this alias in the originating project

    • +

  • source_url – The URL as specified by original source YAML, excluding the alias

    • +

  • extra_data – An optional extra dictionary to return additional data

    • +
+
+
+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.storage.directory.html b/buildstream.storage.directory.html new file mode 100644 index 000000000..e0374e5eb --- /dev/null +++ b/buildstream.storage.directory.html @@ -0,0 +1,595 @@ + + + + + + + + + Directory - Interfacing with files — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Directory - Interfacing with files

+

The Directory class is given to plugins by way of the Sandbox +and in some other instances. This API allows plugins to interface with files +and directory hierarchies owned by BuildStream.

+
+

Paths

+

The path argument to directory functions depict a relative path. Path elements are +separated with the / character regardless of the platform. Both . and .. entries +are permitted. Absolute paths are not permitted, as such it is illegal to specify a path +with a leading / character.

+

Directory objects represent a directory entry within the context of a directory tree, +and the directory returned by +Sandbox.get_virtual_directory() +is the root of the sandbox’s directory tree. Attempts to escape the root of a directory tree +using .. entries will not result in an error, instead .. entries which cross the +root boundary will be evaluated as the root directory. This behavior matches POSIX behavior +of filesystem root directories.

+
+
+
+
+exception DirectoryError(message: str, reason: str = None)
+

Bases: BstError

+

Raised by Directory functions.

+

It is recommended to handle this error and raise a more descriptive +user facing ElementError or SourceError from this error.

+

If this is not handled, the BuildStream core will fail the current +task where the error occurs and present the user with the error.

+
+ +
+
+class FileType(value)
+

Bases: FastEnum

+

Depicts the type of a file

+
+
+DIRECTORY: int = <buildstream.storage.directory.FileType object>
+

A directory

+
+ +
+
+REGULAR_FILE: int = <buildstream.storage.directory.FileType object>
+

A regular file

+
+ +
+ +

A symbolic link

+
+ +
+ +
+
+class FileStat(file_type: int, *, executable: bool = False, size: int = 0, mtime: float = 1321009871)
+

Bases: object

+

Depicts stats about a file

+
+

Note

+

This object can be compared with the equality operator, two FileStat +objects will be considered equivalent if they have the same FileType +and have equivalent attributes.

+
+
+
+file_type: int
+

The FileType of this file

+
+ +
+
+executable: bool
+

Whether this file is executable

+
+ +
+
+size: int
+

The size of the file in bytes

+
+ +
+
+mtime: float
+

The modification time of the file

+
+ +
+ +
+
+class Directory(external_directory=None)
+

Bases: object

+

The Directory object represents a directory in a filesystem hierarchy

+
+

Tip

+

Directory objects behave as a collection of entries in the pythonic sense. +Iterating over a directory will yield the entries, and a directory is +truthy if it contains any entries and falsy if it is empty.

+
+
+
+open_directory(path: str, *, create: bool = False, follow_symlinks: bool = False) → Directory
+

Open a Directory object relative to this directory

+
+
Parameters:
+
    +

  • path – A path relative to this directory.

    • +

  • create – If this is true, the directories will be created if +they don’t already exist.

    • +
+
+
Returns:
+

A Directory object representing the found directory.

+
+
Raises:
+

DirectoryError – if any of the components in subdirectory_spec + cannot be found, or are files, or symlinks to files.

+
+
+
+ +
+
+import_files(external_pathspec: Directory | str, *, filter_callback: Callable[[str], bool] | None = None, collect_result: bool = True) → FileListResult | None
+

Imports some or all files from external_path into this directory.

+
+
Parameters:
+
    +

  • external_pathspec – Either a string containing a pathname, or a +Directory object, to use as the source.

    • +

  • filter_callback – Optional filter callback. Called with the +relative path as argument for every file in the source directory. +The file is imported only if the callable returns True. +If no filter callback is specified, all files will be imported.

    • +

  • collect_result – Whether to collect data for the FileListResult, defaults to True.

    • +
+
+
Returns:
+

A FileListResult report of files imported and overwritten, +or None if collect_result is False.

+
+
Raises:
+

DirectoryError – if any system error occurs.

+
+
+
+ +
+
+import_single_file(external_pathspec: str) → FileListResult
+

Imports a single file from an external path

+
+
Parameters:
+
    +

  • external_pathspec – A string containing a pathname.

    • +

  • properties – Optional list of strings representing file properties to capture when importing.

    • +
+
+
Returns:
+

A FileListResult report of files imported and overwritten.

+
+
Raises:
+

DirectoryError – if any system error occurs.

+
+
+
+ +
+
+export_to_tar(tarfile: TarFile, destination_dir: str, mtime: int = 1321009871) → None
+

Exports this directory into the given tar file.

+
+
Parameters:
+
    +

  • tarfile – A Python TarFile object to export into.

    • +

  • destination_dir – The prefix for all filenames inside the archive.

    • +

  • mtime – mtimes of all files in the archive are set to this.

    • +
+
+
Raises:
+

DirectoryError – if any system error occurs.

+
+
+
+ +
+
+list_relative_paths() → Iterator[str]
+

Generate a list of all relative paths in this directory.

+
+
Yields:
+

All files in this directory with relative paths.

+
+
+
+ +
+
+exists(path: str, *, follow_symlinks: bool = False) → bool
+

Check whether the specified path exists.

+
+
Parameters:
+
    +

  • path – A path relative to this directory.

    • +

  • follow_symlinks – True to follow symlinks.

    • +
+
+
Returns:
+

True if the path exists, False otherwise.

+
+
+
+ +
+
+stat(path: str, *, follow_symlinks: bool = False) → FileStat
+

Get the status of a file.

+
+
Parameters:
+
    +

  • path – A path relative to this directory.

    • +

  • follow_symlinks – True to follow symlinks.

    • +
+
+
Returns:
+

A FileStat object.

+
+
Raises:
+

DirectoryError – if any system error occurs.

+
+
+
+ +
+
+open_file(path: str, *, mode: str = 'r') → Iterator[IO]
+

Open file and return a corresponding file object. In text mode, +UTF-8 is used as encoding.

+
+
Parameters:
+
    +

  • path – A path relative to this directory.

    • +

  • mode (str) – An optional string that specifies the mode in which the file is opened.

    • +
+
+
Yields:
+

The file object for the open file

+
+
Raises:
+

DirectoryError – if any system error occurs.

+
+
+
+ +
+
+file_digest(path: str) → str
+

Return a unique digest of a file.

+
+
Parameters:
+

path – A path relative to this directory.

+
+
Raises:
+

DirectoryError – if the specified path depicts an entry that is not a + FileType.REGULAR_FILE, or if any system error occurs.

+
+
+
+ +
+ +

Return a string representing the path to which the symbolic link points.

+
+
Parameters:
+

path – A path relative to this directory.

+
+
Returns:
+

The path to which the symbolic link points to.

+
+
Raises:
+

DirectoryError – if any system error occurs.

+
+
+
+ +
+
+remove(path: str, *, recursive: bool = False) → None
+

Remove a file, symlink or directory. Symlinks are not followed.

+
+
Parameters:
+
    +

  • path – A path relative to this directory.

    • +

  • recursive – True to delete non-empty directories.

    • +
+
+
Raises:
+

DirectoryError – if any system error occurs.

+
+
+
+ +
+
+rename(src: str, dest: str) → None
+

Rename a file, symlink or directory. If destination path exists +already and is a file or empty directory, it will be replaced.

+
+
Parameters:
+
    +

  • src – A source path relative to this directory.

    • +

  • dest – A destination path relative to this directory.

    • +
+
+
Raises:
+

DirectoryError – if any system error occurs.

+
+
+
+ +
+
+isfile(path: str, *, follow_symlinks: bool = False) → bool
+

Check whether the specified path is an existing regular file.

+
+
Parameters:
+
    +

  • path – A path relative to this directory.

    • +

  • follow_symlinks – True to follow symlinks.

    • +
+
+
Returns:
+

True if the path is an existing regular file, False otherwise.

+
+
+
+ +
+
+isdir(path: str, *, follow_symlinks: bool = False) → bool
+

Check whether the specified path is an existing directory.

+
+
Parameters:
+
    +

  • path – A path relative to this directory.

    • +

  • follow_symlinks – True to follow symlinks.

    • +
+
+
Returns:
+

True if the path is an existing directory, False otherwise.

+
+
+
+ +
+ +

Check whether the specified path is an existing symlink.

+
+
Parameters:
+
    +

  • path – A path relative to this directory.

    • +

  • follow_symlinks – True to follow symlinks.

    • +
+
+
Returns:
+

True if the path is an existing symlink, False otherwise.

+
+
+
+ +
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.storage.html b/buildstream.storage.html new file mode 100644 index 000000000..221655343 --- /dev/null +++ b/buildstream.storage.html @@ -0,0 +1,175 @@ + + + + + + + + + buildstream.storage package — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+ +
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.types.html b/buildstream.types.html new file mode 100644 index 000000000..c94222252 --- /dev/null +++ b/buildstream.types.html @@ -0,0 +1,331 @@ + + + + + + + + + Foundation types — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Foundation types

+
+
+
+class FastEnum(value)
+

Bases: object

+

A reimplementation of a subset of the Enum functionality, which is far quicker than Enum.

+

enum.Enum attributes accesses can be really slow, and slow down the execution noticeably. +This reimplementation doesn’t suffer the same problems, but also does not reimplement everything.

+
+
+name = None
+

The name of the current Enum entry, same as enum.Enum.name()

+
+ +
+
+value = None
+

The value of the current Enum entry, same as enum.Enum.value()

+
+ +
+
+classmethod values()
+

Get all the possible values for the enum.

+
+
Returns:
+

the list of all possible values for the enum

+
+
Return type:
+

list

+
+
+
+ +
+ +
+
+class CoreWarnings
+

Bases: object

+

Some common warnings which are raised by core functionalities within BuildStream are found in this class.

+
+
+OVERLAPS = 'overlaps'
+

This warning will be produced when buildstream detects an overlap on an element +which is not whitelisted. See Overlap Whitelist

+
+ +
+
+UNSTAGED_FILES = 'unstaged-files'
+

This warning will be produced when a file cannot be staged. This can happen when +a file overlaps with a directory in the sandbox that is not empty.

+
+ +
+
+REF_NOT_IN_TRACK = 'ref-not-in-track'
+

This warning will be produced when a source is configured with a reference +which is found to be invalid based on the configured track

+
+ +
+
+UNALIASED_URL = 'unaliased-url'
+

A URL used for fetching a sources was specified without specifying any +alias

+
+ +
+ +
+
+class OverlapAction
+

Bases: FastEnum

+

Defines what action to take when files staged into the sandbox overlap.

+
+

Note

+

This only dictates what happens when functions such as +Element.stage_artifact() and +Element.stage_dependency_artifacts() +are called multiple times in an Element’s Element.stage() +implementation, and the files staged from one function call result in overlapping files staged +from previous invocations.

+

If multiple staged elements overlap eachother within a single call to +Element.stage_dependency_artifacts(), +then the overlap whitelist will be ovserved, and warnings will +be issued for overlapping files, which will be fatal warnings if +CoreWarnings.OVERLAPS is specified +as a fatal warning.

+
+
+
+ERROR = <buildstream.types.OverlapAction object>
+

It is an error to overlap previously staged files

+
+ +
+
+WARNING = <buildstream.types.OverlapAction object>
+

A warning will be issued for previously staged files, which will fatal if +CoreWarnings.OVERLAPS is specified +as a fatal warning in the project.

+
+ +
+
+IGNORE = <buildstream.types.OverlapAction object>
+

Overlapping files are acceptable, and do not cause any warning or error.

+
+ +
+ +
+
+SourceRef
+

A simple python object used to describe and exact set of sources

+

This can be None in order to represent an absense of a source reference, +otherwise it can be int, str, or a complex list or dict consisting +of int, str, list and dict types.

+

The order of elements in list objects is meaningful and should be produced +deterministically by Source implementations, as this order will effect +cache keys.

+

See the source documentation for more detils on how +Source implementations are expected to handle the source ref.

+

alias of None | int | str | List[Any] | Dict[str, Any]

+
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/buildstream.utils.html b/buildstream.utils.html new file mode 100644 index 000000000..fd2ddf57e --- /dev/null +++ b/buildstream.utils.html @@ -0,0 +1,558 @@ + + + + + + + + + Utilities — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Utilities

+
+
+
+exception UtilError(message, reason=None)
+

Bases: BstError

+

Raised by utility functions when system calls fail.

+

This will be handled internally by the BuildStream core, +if you need to handle this error, then it should be reraised, +or either of the ElementError or SourceError +exceptions should be raised from this error.

+
+ +
+
+exception ProgramNotFoundError(message, reason=None)
+

Bases: BstError

+

Raised if a required program is not found.

+

It is normally unneeded to handle this exception from plugin code.

+
+ +
+
+exception DirectoryExistsError
+

Bases: OSError

+

Raised when a os.rename is attempted but the destination is an existing directory.

+
+ +
+
+class FileListResult
+

Bases: object

+

An object which stores the result of one of the operations +which run on a list of files.

+
+
+overwritten
+

List of files which were overwritten in the target directory

+
+ +
+
+ignored
+

List of files which were ignored, because they would have +replaced a non empty directory

+
+ +
+
+failed_attributes
+

List of files for which attributes could not be copied over

+
+ +
+
+files_written
+

List of files that were written.

+
+ +
+ +
+
+list_relative_paths(directory: str) → Iterator[str]
+

A generator for walking directory relative paths

+

This generator is useful for checking the full manifest of +a directory.

+

Symbolic links will not be followed, but will be included +in the manifest.

+
+
Parameters:
+

directory – The directory to list files in

+
+
Yields:
+

Relative filenames in directory

+
+
+
+ +
+
+glob(paths: Iterable[str], pattern: str) → Iterator[str]
+

A generator to yield paths which match the glob pattern

+
+
Parameters:
+
    +

  • paths (iterable) – The paths to check

    • +

  • pattern (str) – A glob pattern

    • +
+
+
+

This generator will iterate over the passed paths and +yield only the filenames which matched the provided pattern.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Meta

Description

*

Zero or more of any character, excepting path separators

**

Zero or more of any character, including path separators

?

One of any character, except for path separators

[abc]

One of any of the specified characters

[a-z]

One of the characters in the specified range

[!abc]

Any single character, except the specified characters

[!a-z]

Any single character, except those in the specified range

+
+

Note

+

Escaping of the metacharacters is not possible

+
+
+ +
+
+sha256sum(filename: str) → str
+

Calculate the sha256sum of a file

+
+
Parameters:
+

filename – A path to a file on disk

+
+
Returns:
+

An sha256 checksum string

+
+
Raises:
+

UtilError – In the case there was an issue opening + or reading filename

+
+
+
+ +
+
+safe_copy(src: str, dest: str, *, copystat: bool = True, result: FileListResult | None = None) → None
+

Copy a file while optionally preserving attributes

+
+
Parameters:
+
    +

  • src – The source filename

    • +

  • dest – The destination filename

    • +

  • copystat – Whether to preserve attributes

    • +

  • result – An optional collective result

    • +
+
+
Raises:
+

UtilError – In the case of unexpected system call failures

+
+
+

This is almost the same as shutil.copy2() when copystat is True, +except that we unlink dest before overwriting it if it exists, just +incase dest is a hardlink to a different file.

+
+ +
+ +

Try to create a hardlink, but resort to copying in the case of cross device links.

+
+
Parameters:
+
    +

  • src – The source filename

    • +

  • dest – The destination filename

    • +

  • result – An optional collective result

    • +
+
+
Raises:
+

UtilError – In the case of unexpected system call failures

+
+
+
+ +
+
+safe_remove(path: str) → bool
+

Removes a file or directory

+

This will remove a file if it exists, and will +remove a directory if the directory is empty.

+
+
Parameters:
+

path – The path to remove

+
+
Returns:
+

True if path was removed or did not exist, False +if path was a non empty directory.

+
+
Raises:
+

UtilError – In the case of unexpected system call failures

+
+
+
+ +
+
+copy_files(src: str, dest: str, *, filter_callback: Callable[[str], bool] | None = None, ignore_missing: bool = False, report_written: bool = False) → FileListResult
+

Copy files from source to destination.

+
+
Parameters:
+
    +

  • src – The source directory

    • +

  • dest – The destination directory

    • +

  • filter_callback – Optional filter callback. Called with the relative path as +argument for every file in the source directory. The file is +copied only if the callable returns True. If no filter callback +is specified, all files will be copied.

    • +

  • ignore_missing – Dont raise any error if a source file is missing

    • +

  • report_written – Add to the result object the full list of files written

    • +
+
+
Returns:
+

The result describing what happened during this file operation

+
+
Raises:
+

UtilError – In the case of unexpected system call failures

+
+
+
+

Note

+

Directories in dest are replaced with files from src, +unless the existing directory in dest is not empty in which +case the path will be reported in the return value.

+

UNIX domain socket files from src are ignored.

+
+
+ +
+ +

Hardlink files from source to destination.

+
+
Parameters:
+
    +

  • src – The source directory

    • +

  • dest – The destination directory

    • +

  • filter_callback – Optional filter callback. Called with the relative path as +argument for every file in the source directory. The file is +hardlinked only if the callable returns True. If no filter +callback is specified, all files will be hardlinked.

    • +

  • ignore_missing – Dont raise any error if a source file is missing

    • +

  • report_written – Add to the result object the full list of files written

    • +
+
+
Returns:
+

The result describing what happened during this file operation

+
+
Raises:
+

UtilError – In the case of unexpected system call failures

+
+
+
+

Note

+

Directories in dest are replaced with files from src, +unless the existing directory in dest is not empty in which +case the path will be reported in the return value.

+
+
+

Note

+

If a hardlink cannot be created due to crossing filesystems, +then the file will be copied instead.

+

UNIX domain socket files from src are ignored.

+
+
+ +
+
+get_host_tool(name: str) → str
+

Get the full path of a host tool

+
+
Parameters:
+

name (str) – The name of the program to search for

+
+
Returns:
+

The full path to the program, if found

+
+
Raises:
+

.ProgramNotFoundError –

+
+
+
+ +
+
+get_bst_version() → Tuple[int, int]
+

Gets the major, minor release portion of the +BuildStream version.

+
+
Returns:
+

A 2-tuple of form (major version, minor version)

+
+
+
+ +
+
+move_atomic(source: Path | str, destination: Path | str, *, ensure_parents: bool = True) → None
+

Move the source to the destination using atomic primitives.

+

This uses os.rename to move a file or directory to a new destination. +It wraps some OSError thrown errors to ensure their handling is correct.

+

The main reason for this to exist is that rename can throw different errors +for the same symptom (https://www.unix.com/man-page/POSIX/3posix/rename/) +when we are moving a directory.

+

We are especially interested here in the case when the destination already +exists, is a directory and is not empty. In this case, either EEXIST or +ENOTEMPTY can be thrown.

+

In order to ensure consistent handling of these exceptions, this function +should be used instead of os.rename

+
+
Parameters:
+
    +

  • source – source to rename

    • +

  • destination – destination to which to move the source

    • +

  • ensure_parents – Whether or not to create the parent’s directories +of the destination (default: True)

    • +
+
+
Raises:
+
    +

  • DirectoryExistsError – if the destination directory already exists and is + not empty

    • +

  • OSError – if another filesystem level error occured

    • +
+
+
+
+ +
+
+save_file_atomic(filename: str, mode: str = 'w', *, buffering: int = -1, encoding: str | None = None, errors: str | None = None, newline: str | None = None, closefd: bool = True, opener: Callable[[str, int], int] | None = None, tempdir: str | None = None) → Iterator[IO]
+

Save a file with a temporary name and rename it into place when ready.

+

This is a context manager which is meant for saving data to files. +The data is written to a temporary file, which gets renamed to the target +name when the context is closed. This avoids readers of the file from +getting an incomplete file.

+

Example:

+
with save_file_atomic('/path/to/foo', 'w') as f:
+    f.write(stuff)
+
+
+

The file will be called something like tmpCAFEBEEF until the +context block ends, at which point it gets renamed to foo. The +temporary file will be created in the same directory as the output file. +The filename parameter must be an absolute path.

+

If an exception occurs or the process is terminated, the temporary file will +be deleted.

+
+ +
+
+get_umask()
+
+ + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/core_additional.html b/core_additional.html new file mode 100644 index 000000000..c86c85071 --- /dev/null +++ b/core_additional.html @@ -0,0 +1,135 @@ + + + + + + + + + Additional writings — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Additional writings

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/core_format.html b/core_format.html new file mode 100644 index 000000000..084750e07 --- /dev/null +++ b/core_format.html @@ -0,0 +1,172 @@ + + + + + + + + + Project format — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+ +
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/core_framework.html b/core_framework.html new file mode 100644 index 000000000..cc99e2293 --- /dev/null +++ b/core_framework.html @@ -0,0 +1,257 @@ + + + + + + + + + Plugin API reference — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Plugin API reference

+

The core public APIs are of interest to anyone who wishes to +implement custom Element or +Source plugins, and can also be +useful for working on BuildStream itself.

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/core_plugins.html b/core_plugins.html new file mode 100644 index 000000000..6780c5a0e --- /dev/null +++ b/core_plugins.html @@ -0,0 +1,189 @@ + + + + + + + + + Plugin specific documentation — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Plugin specific documentation

+

Plugins provide their own individual plugin specific YAML configurations, +The element .bst files can specify plugin specific configuration in +the config section, while sources declared on a +given element specify their plugin specific configuration directly +in their source declarations.

+
+

Elements

+
+ +
+
+
+

Sources

+

All source plugins can be staged into an arbitrary directory within the build +sandbox with the directory option. +See Source class built-in functionality for more +information.

+
+ +
+
+
+

External plugins

+

External plugins need to be loading through junctions, +or alternatively installed separately in the python environment where you are +running BuildStream and loaded using the pip method.

+

Here is a list of BuildStream plugin projects known to us at this time:

+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/developing/strict-mode.html b/developing/strict-mode.html new file mode 100644 index 000000000..fc890038b --- /dev/null +++ b/developing/strict-mode.html @@ -0,0 +1,738 @@ + + + + + + + + + 2. Strict mode — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

2. Strict mode

+

In this section, we will cover the usage of strict vs non-strict +build plans in conjunction with workspaces, and how this +can help to improve your edit/compile/test cycles.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/strict-mode +subdirectory.

+
+
+

2.1. Overview

+

When working with BuildStream to create integrations, it is typical that you have a +lot of components to build, and you frequently need to modify a component +at various levels of the stack. When developing one or more applications, you might +want to open a workspace and fix a bug in an application, or you might need to +open a workspace on a low level shared library to fix the behavior of one or +more misbehaving applications.

+

By default, BuildStream will always choose to be deterministic in order to +produce the most correct build results as possible. As such, modifying a low +level library will result in rebuilding all of it’s reverse dependencies, but +this can be very time consuming and inconvenient for your edit/compile/test +cycles.

+

This is when enabling non-strict build plans +can be helpful.

+

To illustrate the facets of how this works, this example will present a project +consisting of an application which is linked both statically and dynamically +linked to a common library.

+
+
+

2.2. Project structure

+

This project is mostly based on the integration commands +example, as such we will ignore large parts of this project and only focus +on the elements which are of specific interest.

+

To illustrate the relationship of these two applications and the library, +let’s briefly take a look at the underlying Makefiles which are used in this +project, starting with the library and followed by both Makefiles used to +build the application.

+
+

2.2.1. files/libhello/Makefile

+
# Sample makefile for hello library
+#
+.PHONY: all install
+
+all: libhello.so libhello.a
+
+install: all
+	install -d ${DESTDIR}${PREFIX}/lib
+	install -d ${DESTDIR}${PREFIX}/include
+	install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
+	install -m 644 libhello.a ${DESTDIR}${PREFIX}/lib
+	install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
+
+%.o: %.c %.h
+	$(CC) -c $< -o $@ -Wall
+
+libhello.a: libhello.o
+	$(AR) rcs $@ $^
+
+libhello.so: libhello.o
+	$(CC) -shared -o $@ $<
+
+
+
+
+

2.2.2. files/hello/Makefile.dynamic

+
# Sample makefile for hello.c
+#
+.PHONY: all install
+
+all: hello
+
+install: all
+	install -d ${DESTDIR}${PREFIX}/bin
+	install -m 755 hello ${DESTDIR}${PREFIX}/bin
+
+hello: hello.c
+	$(CC) -Wall -o $@ $< -lhello
+
+
+
+
+

2.2.3. files/hello/Makefile.static

+
# Sample makefile for hello.c
+#
+.PHONY: all install
+
+all: hello
+
+install: all
+	install -d ${DESTDIR}${PREFIX}/bin
+	install -m 755 hello ${DESTDIR}${PREFIX}/bin
+
+hello: hello.c
+	$(CC) -Wall -o $@ $< /usr/lib/libhello.a
+
+
+

As we can see, we have a library that is distributed both as the dynamic +library libhello.so and also as the static archive libhello.a.

+

Now let’s take a look at the two separate elements which build the +application, first the dynamically linked version and then the static one.

+
+
+

2.2.4. elements/hello-dynamic.bst

+
kind: manual
+description: |
+
+  The dynamically linked hello application
+
+# Depend on the hello library
+depends:
+- libhello.bst
+
+# Stage the files/hello directory for building
+sources:
+  - kind: local
+    path: files/hello
+
+# Now configure the commands to run
+config:
+
+  build-commands:
+  - make -f Makefile.dynamic PREFIX="%{prefix}"
+
+  install-commands:
+  - make -f Makefile.dynamic -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
+
+
+

Nothing very special to observe about this hello program, just a +manual element quite similar to the one we’ve +already seen in the running commands +example.

+
+
+

2.2.5. elements/hello-static.bst

+
kind: manual
+description: |
+
+  The statically linked hello application
+
+# Depend on the hello library with the strict option
+#
+depends:
+- filename: libhello.bst
+  strict: true
+
+# Stage the files/hello directory for building
+sources:
+  - kind: local
+    path: files/hello
+
+# Now configure the commands to run
+config:
+
+  build-commands:
+  - make -f Makefile.static PREFIX="%{prefix}"
+
+  install-commands:
+  - make -f Makefile.static -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
+
+
+

Almost the same as the dynamic element, except here we have declared the +dependency to the libhello.bst element differently: this time we have enabled +the strict option in the dependency declaration.

+

The side effect of setting this option is that hello-static.bst will be +rebuilt any time that libhello.bst has changed, even when +non-strict build plans have been enabled.

+
+

Tip

+

Some element plugins are designed to consume the content of their +dependencies entirely, and output an artifact without any transient +runtime dependencies, an example of this is the compose +element.

+

In cases such as compose, it is not necessary to +explicitly annotate their dependencies as strict.

+

It is only helpful to set the strict attribute on a +dependency declaration in the case that the +specific dependency relationship causes data to be consumed verbatim, +as is the case with static linking.

+
+
+
+
+

2.3. Using the project

+

For the sake of brevity, let’s assume that you’ve already built all of the +elements of this project, and that you want to make some changes to the +libhello.bst element, and test how it might effect the hello program.

+
+

2.3.1. Everything is already built

+ +
+user@host:~/strict-mode$ bst show hello-static.bst hello-dynamic.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888279.821189    7192 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+      cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst 
+      cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst 
+      cached cfc97a856b683a50e0b18ee6073c6283763d6ccc3937717d60f4b7281fa5e562 libhello.bst 
+      cached 27d2a1506044342dd91b6cfd0ca7ca98b5ed14d9c500b8e8beb9b2202dfb21db hello-static.bst 
+      cached 4aacf94bf921018c2224754cdba580590bcdcd26a293bc770f4fd08152369bf4 hello-dynamic.bst 
+
+
+
+

2.3.2. Open a workspace and modify libhello.c

+

Now let’s open up a workspace on the hello library

+ +
+user@host:~/strict-mode$ bst workspace open --directory workspace_libhello libhello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888280.224801    7220 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][        ][    main:core activity                 ] STATUS  Creating workspace for element libhello.bst
+[--:--:--][cfc97a85][    main:libhello.bst                  ] START   Staging sources to /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
+[--:--:--][        ][    main:libhello.bst                  ] START   Staging local files into CAS
+[00:00:00][        ][    main:libhello.bst                  ] SUCCESS Staging local files into CAS
+[00:00:00][cfc97a85][    main:libhello.bst                  ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
+[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element libhello.bst
+
+

And go ahead and make a modification like this:

+
--- libhello.c
++++ libhello.c
+@@ -5,5 +5,5 @@
+ 
+ void hello(const char *person)
+ {
+-  printf("Hello %s\n", person);
++  printf("Good morning %s\n", person);
+ }
+
+
+
+
+

2.3.3. Observing hello-dynamic.bst

+

Let’s take a look at the bst show output for +the dynamically linked hello-dynamic.bst element.

+ +
+user@host:~/strict-mode$ bst show hello-dynamic.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888280.625510    7249 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+      cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst 
+      cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst 
+   buildable 825d9815607a5d73945c9f1fd457834e02870a54a03bf09619bb754763e24fc8 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
+     waiting 05bb664034f1bf5b24ab99003b7e0edaa4ed9a890d67eb4677cd0edf81fd1d7b hello-dynamic.bst 
+
+

As one might expect, the libhello.bst element is ready to be built +after having been modified, and the hello-dynamic.bst element is +waiting for libhello.bst to be built before it can build.

+

Now let’s take a look at the same elements if we pass the --no-strict +option to bst:

+ +
+user@host:~/strict-mode$ bst --no-strict show hello-dynamic.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888281.024279    7277 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+      cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst 
+      cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst 
+   buildable 825d9815607a5d73945c9f1fd457834e02870a54a03bf09619bb754763e24fc8 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
+      cached 4aacf94bf921018c2224754cdba580590bcdcd26a293bc770f4fd08152369bf4 hello-dynamic.bst 
+
+

Note that this time, the libhello.bst still needs to be built, +but the hello-dymamic.bst element is showing up as cached.

+
+

Tip

+

The bst show output will show some cache +keys dimmed out in the case that they are not entirely deterministic.

+

Here we can see that hello-dynamic.bst is dimmed out because +it will not be rebuilt against the changed libhello.bst element, +and it also has a different cache key because of this.

+
+
+
+

2.3.4. Observing hello-static.bst

+

Now let’s observe the hello-static.bst element with strict mode +disabled:

+ +
+user@host:~/strict-mode$ bst --no-strict show hello-static.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888281.427621    7306 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+      cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst 
+      cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst 
+   buildable 825d9815607a5d73945c9f1fd457834e02870a54a03bf09619bb754763e24fc8 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
+     waiting abdc7cb10e1696d9b3c51baf21e83d847f7164f84b3a4627b40bf1ff50322330 hello-static.bst 
+
+

Note that in this case the hello-strict.bst is going to be +rebuilt even in strict mode. This is because we annotated the +declaration of the libhello.bst dependency with the strict +attribute.

+

We did this because hello-strict.bst consumes the input of +libhello.bst verbatim, by way of statically linking to it, instead +of merely being affected by the content of libhello.bst at runtime, +as would be the case of static linking.

+
+
+

2.3.5. Building and running hello-dynamic.bst

+

Now let’s build hello-dynamic.bst with strict mode disabled.

+ +
+user@host:~/strict-mode$ bst --no-strict build hello-dynamic.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888281.829086    7334 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:18:01
+    Project:       strict-mode (/home/user/buildstream/doc/examples/strict-mode)
+    Targets:       hello-dynamic.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-bho1gvzz/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-bho1gvzz
+    Log Files:               /home/user/buildstream/doc/run-bst-bho1gvzz/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-bho1gvzz/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-bho1gvzz/build
+    Strict Build Plan:       No
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: strict-mode
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        local:     core plugin
+        workspace: core plugin
+        tar:       core plugin
+
+Pipeline
+      cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst 
+      cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst 
+   buildable 825d9815607a5d73945c9f1fd457834e02870a54a03bf09619bb754763e24fc8 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
+      cached 4aacf94bf921018c2224754cdba580590bcdcd26a293bc770f4fd08152369bf4 hello-dynamic.bst 
+===============================================================================
+[--:--:--][825d9815][   fetch:libhello.bst                  ] START   strict-mode/libhello/825d9815-fetch.20241106-101801.log
+[00:00:00][825d9815][   fetch:libhello.bst                  ] SUCCESS strict-mode/libhello/825d9815-fetch.20241106-101801.log
+[--:--:--][825d9815][   build:libhello.bst                  ] START   strict-mode/libhello/825d9815-build.20241106-101801.log
+[--:--:--][825d9815][   build:libhello.bst                  ] START   Staging dependencies at: /
+[00:00:00][825d9815][   build:libhello.bst                  ] SUCCESS Staging dependencies at: /
+[--:--:--][825d9815][   build:libhello.bst                  ] START   Integrating sandbox
+[00:00:00][825d9815][   build:libhello.bst                  ] SUCCESS Integrating sandbox
+[--:--:--][825d9815][   build:libhello.bst                  ] START   Staging sources
+[00:00:00][825d9815][   build:libhello.bst                  ] SUCCESS Staging sources
+[--:--:--][825d9815][   build:libhello.bst                  ] START   Running commands
+
+    make PREFIX="/usr"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][825d9815][   build:libhello.bst                  ] SUCCESS Running commands
+[--:--:--][825d9815][   build:libhello.bst                  ] START   Caching artifact
+[00:00:00][825d9815][   build:libhello.bst                  ] SUCCESS Caching artifact
+[00:00:00][825d9815][   build:libhello.bst                  ] SUCCESS strict-mode/libhello/825d9815-build.20241106-101801.log
+[00:00:01][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       4
+    Session:     4
+    Fetch Queue: processed 1, skipped 3, failed 0 
+    Build Queue: processed 1, skipped 3, failed 0
+
+

Note that the bst build command completed without +having to build hello-dynamic.bst at all.

+

And now we can also run hello-dynamic.bst

+ +
+user@host:~/strict-mode$ bst --no-strict shell hello-dynamic.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888283.144630    7414 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][4aacf94b][    main:hello-dynamic.bst             ] START   Staging dependencies
+[00:00:00][4aacf94b][    main:hello-dynamic.bst             ] SUCCESS Staging dependencies
+[--:--:--][4aacf94b][    main:hello-dynamic.bst             ] START   Integrating sandbox
+[--:--:--][        ][    main:hello-dynamic.bst             ] START   Running commands
+
+    ldconfig "/usr/lib"
+
++ sh -e -c ldconfig "/usr/lib"
+[00:00:00][        ][    main:hello-dynamic.bst             ] SUCCESS Running commands
+[00:00:00][4aacf94b][    main:hello-dynamic.bst             ] SUCCESS Integrating sandbox
+[--:--:--][4aacf94b][    main:hello-dynamic.bst             ] STATUS  Running command
+
+    hello
+
+Good morning stranger
+
+

When running hello-dynamic.bst with no-strict mode, we are +actually reusing the old build of hello-dynamic.bst staged against +the new build of the modified libhello.bst element.

+
+
+

2.3.6. Building and running hello-static.bst

+

Finally, if we build hello-static.bst with strict mode disabled, +we can see that it will be rebuilt regardless of strict mode being +enabled.

+ +
+user@host:~/strict-mode$ bst --no-strict build hello-static.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888284.349311    7478 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:18:04
+    Project:       strict-mode (/home/user/buildstream/doc/examples/strict-mode)
+    Targets:       hello-static.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-bho1gvzz/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-bho1gvzz
+    Log Files:               /home/user/buildstream/doc/run-bst-bho1gvzz/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-bho1gvzz/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-bho1gvzz/build
+    Strict Build Plan:       No
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: strict-mode
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        local:     core plugin
+        workspace: core plugin
+        tar:       core plugin
+
+Pipeline
+      cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst 
+      cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst 
+      cached 825d9815607a5d73945c9f1fd457834e02870a54a03bf09619bb754763e24fc8 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
+   buildable abdc7cb10e1696d9b3c51baf21e83d847f7164f84b3a4627b40bf1ff50322330 hello-static.bst 
+===============================================================================
+[--:--:--][abdc7cb1][   build:hello-static.bst              ] START   strict-mode/hello-static/abdc7cb1-build.20241106-101804.log
+[--:--:--][abdc7cb1][   build:hello-static.bst              ] START   Staging dependencies at: /
+[00:00:00][abdc7cb1][   build:hello-static.bst              ] SUCCESS Staging dependencies at: /
+[--:--:--][abdc7cb1][   build:hello-static.bst              ] START   Integrating sandbox
+[00:00:00][abdc7cb1][   build:hello-static.bst              ] SUCCESS Integrating sandbox
+[--:--:--][abdc7cb1][   build:hello-static.bst              ] START   Staging sources
+[00:00:00][abdc7cb1][   build:hello-static.bst              ] SUCCESS Staging sources
+[--:--:--][abdc7cb1][   build:hello-static.bst              ] START   Running commands
+
+    make -f Makefile.static PREFIX="/usr"
+    make -f Makefile.static -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][abdc7cb1][   build:hello-static.bst              ] SUCCESS Running commands
+[--:--:--][abdc7cb1][   build:hello-static.bst              ] START   Caching artifact
+[00:00:00][abdc7cb1][   build:hello-static.bst              ] SUCCESS Caching artifact
+[00:00:00][abdc7cb1][   build:hello-static.bst              ] SUCCESS strict-mode/hello-static/abdc7cb1-build.20241106-101804.log
+[00:00:00][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       4
+    Session:     4
+    Fetch Queue: processed 0, skipped 4, failed 0 
+    Build Queue: processed 1, skipped 3, failed 0
+
+

This is of course because we declared its dependency on libhello.bst +as a strict dependency.

+

And by the same virtue, we can see that when we run the example +it has properly relinked against the changed static archive, and +has the updated text in the greeting:

+ +
+user@host:~/strict-mode$ bst --no-strict shell hello-static.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888285.553760    7553 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][abdc7cb1][    main:hello-static.bst              ] START   Staging dependencies
+[00:00:00][abdc7cb1][    main:hello-static.bst              ] SUCCESS Staging dependencies
+[--:--:--][abdc7cb1][    main:hello-static.bst              ] START   Integrating sandbox
+[--:--:--][        ][    main:hello-static.bst              ] START   Running commands
+
+    ldconfig "/usr/lib"
+
++ sh -e -c ldconfig "/usr/lib"
+[00:00:00][        ][    main:hello-static.bst              ] SUCCESS Running commands
+[00:00:00][abdc7cb1][    main:hello-static.bst              ] SUCCESS Integrating sandbox
+[--:--:--][abdc7cb1][    main:hello-static.bst              ] STATUS  Running command
+
+    hello
+
+Good morning stranger
+
+
+
+
+

2.4. Summary

+

In this chapter we’ve explored how to use non-strict build plans +in order to avoid rebuilding reverse dependencies of a lower level +element you might be working with in a workspace, +consequently improving your edit/compile/test experience.

+

We’ve also explained how to ensure your project still works properly +with non-strict build plans when some elements perform static linking +(or other operations which consume data from their dependencies +verbatim), by annotating dependency declarations +as strict.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/developing/workspaces.html b/developing/workspaces.html new file mode 100644 index 000000000..a74acf984 --- /dev/null +++ b/developing/workspaces.html @@ -0,0 +1,544 @@ + + + + + + + + + 1. Workspaces — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

1. Workspaces

+

In this section we will cover the use of BuildStream’s workspaces feature when +devloping a BuildStream project.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/developing +subdirectory.

+
+

We will start with the project used in the running commands +tutorial. Recall the element hello.bst, which builds the bellow C file:

+
/*
+ * hello.c - Simple hello world program
+ */
+#include <stdio.h>
+
+int main(int argc, char *argv[])
+{
+  printf("Hello World\n");
+  return 0;
+}
+
+
+

Suppose we now want to alter the functionality of the hello command. We can +make changes to the source code of Buildstream elements by making use of +BuildStream’s workspace command.

+
+

1.1. Opening a workspace

+

First we need to open a workspace, we can do this by running

+ +
+user@host:~/developing$ bst workspace open --directory workspace_hello hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888166.326334    1737 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][        ][    main:core activity                 ] STATUS  Creating workspace for element hello.bst
+[--:--:--][a6a39926][    main:hello.bst                     ] START   Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
+[--:--:--][        ][    main:hello.bst                     ] START   Staging local files into CAS
+[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files into CAS
+[00:00:00][a6a39926][    main:hello.bst                     ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
+[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element hello.bst
+
+

This command has created the workspace_hello directory in which you can see +the source for the hello.bst element, i.e. hello.c and the corresponding +makefile.

+

You can view existing workspaces using

+ +
+user@host:~/developing$ bst workspace list
+
+workspaces:
+- element: hello.bst
+  directory: /home/user/buildstream/doc/examples/developing/workspace_hello
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888166.745122    1765 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+
+
+
+

1.2. Making code changes

+

Let’s say we want to alter the message printed when the hello command is run. +We can open workspace_hello/hello.c and make the following change:

+
-- hello.c	2018-06-25 14:48:32.077568920 +0100
++++ hello.c	2018-06-25 14:49:23.025553785 +0100
+@@ -5,6 +5,6 @@
+ 
+ int main(int argc, char *argv[])
+ {
+-  printf("Hello World\n");
++  printf("Hello World\nWe can use workspaces!\n");
+   return 0;
+ }
+
+
+

Now, rebuild the hello.bst element.

+ +
+user@host:~/developing$ bst build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888167.053250    1793 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:16:07
+    Project:       developing (/home/user/buildstream/doc/examples/developing)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-8cg84roo/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-8cg84roo
+    Log Files:               /home/user/buildstream/doc/run-bst-8cg84roo/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-8cg84roo/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-8cg84roo/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: developing
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        workspace: core plugin
+        tar:       core plugin
+
+Pipeline
+fetch needed b65366701aaab8b2f85afa7158f4036a1f55a8150db42fff45e0830cc8388d73 base/alpine.bst 
+     waiting 07db7692b3828f7021ae8c1b0b6f715801e64cf24fcb1621a67b47c110fc3c70 base.bst 
+     waiting 2eeebcc564b86377e5f029c59454441c88e6ce3d48cdb88e358ea249701e8989 hello.bst Workspace: /home/user/buildstream/doc/examples/developing/workspace_hello
+===============================================================================
+[--:--:--][b6536670][   fetch:base/alpine.bst               ] START   developing/base-alpine/b6536670-fetch.20241106-101607.log
+[--:--:--][b6536670][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[--:--:--][07db7692][   fetch:base.bst                      ] START   developing/base/07db7692-fetch.20241106-101607.log
+[--:--:--][2eeebcc5][   fetch:hello.bst                     ] START   developing/hello/2eeebcc5-fetch.20241106-101607.log
+[00:00:00][07db7692][   fetch:base.bst                      ] SUCCESS developing/base/07db7692-fetch.20241106-101607.log
+[00:00:00][2eeebcc5][   fetch:hello.bst                     ] SUCCESS developing/hello/2eeebcc5-fetch.20241106-101607.log
+[00:00:00][b6536670][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:06][b6536670][   fetch:base/alpine.bst               ] SUCCESS developing/base-alpine/b6536670-fetch.20241106-101607.log
+[--:--:--][b6536670][   build:base/alpine.bst               ] START   developing/base-alpine/b6536670-build.20241106-101613.log
+[--:--:--][b6536670][   build:base/alpine.bst               ] START   Staging sources
+[00:00:00][b6536670][   build:base/alpine.bst               ] SUCCESS Staging sources
+[--:--:--][b6536670][   build:base/alpine.bst               ] START   Caching artifact
+[00:00:00][b6536670][   build:base/alpine.bst               ] SUCCESS Caching artifact
+[00:00:00][b6536670][   build:base/alpine.bst               ] SUCCESS developing/base-alpine/b6536670-build.20241106-101613.log
+[--:--:--][07db7692][   build:base.bst                      ] START   developing/base/07db7692-build.20241106-101613.log
+[--:--:--][07db7692][   build:base.bst                      ] START   Caching artifact
+[00:00:00][07db7692][   build:base.bst                      ] SUCCESS Caching artifact
+[00:00:00][07db7692][   build:base.bst                      ] SUCCESS developing/base/07db7692-build.20241106-101613.log
+[--:--:--][2eeebcc5][   build:hello.bst                     ] START   developing/hello/2eeebcc5-build.20241106-101613.log
+[--:--:--][2eeebcc5][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][2eeebcc5][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][2eeebcc5][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][2eeebcc5][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][2eeebcc5][   build:hello.bst                     ] START   Staging sources
+[00:00:00][2eeebcc5][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][2eeebcc5][   build:hello.bst                     ] START   Running commands
+
+    make PREFIX="/usr"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][2eeebcc5][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][2eeebcc5][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][2eeebcc5][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:00][2eeebcc5][   build:hello.bst                     ] SUCCESS developing/hello/2eeebcc5-build.20241106-101613.log
+[00:00:07][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 3, skipped 0, failed 0 
+    Build Queue: processed 3, skipped 0, failed 0
+
+

Note that if you run the command from inside the workspace, the element name is optional.

+ +
+user@host:~/workspace_hello$ bst build
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888174.396038    1853 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:16:14
+    Project:       developing (/home/user/buildstream/doc/examples/developing)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-8cg84roo/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-8cg84roo
+    Log Files:               /home/user/buildstream/doc/run-bst-8cg84roo/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-8cg84roo/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-8cg84roo/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: developing
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        workspace: core plugin
+        tar:       core plugin
+
+Pipeline
+      cached b65366701aaab8b2f85afa7158f4036a1f55a8150db42fff45e0830cc8388d73 base/alpine.bst 
+      cached 07db7692b3828f7021ae8c1b0b6f715801e64cf24fcb1621a67b47c110fc3c70 base.bst 
+      cached 2eeebcc564b86377e5f029c59454441c88e6ce3d48cdb88e358ea249701e8989 hello.bst Workspace: /home/user/buildstream/doc/examples/developing/workspace_hello
+===============================================================================
+[00:00:00][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 0, skipped 3, failed 0 
+    Build Queue: processed 0, skipped 3, failed 0
+
+

Now running the hello command using bst shell:

+ +
+user@host:~/developing$ bst shell hello.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888174.816882    1881 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][2eeebcc5][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][2eeebcc5][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][2eeebcc5][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][2eeebcc5][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][2eeebcc5][    main:hello.bst                     ] STATUS  Running command
+
+    hello
+
+Hello World
+We can use workspaces!
+
+

This gives us the new message we changed in hello.c.

+

From this point we have several options. If the source is under version control +we can commit our changes and push them to the remote repository.

+
+
+

1.3. Incremental builds

+

Once you have opened up your workspace, the workspace build directory will be +reused for subsequent builds, which should improve your edit/compile/test cycle +time when working with an open workspace.

+

In order to optimize incremental builds, BuildStream treats build configure steps +separately from the main build steps, and will only call the +Element.prepare() method on +an element plugin the first time it gets built. This avoids needlessly rebuilding +objects due to header files and such being unconditionally recreated by configuration +scripts (such as the typical ./configure script which is called for autotools +elements for instance).

+

A caveat of this optimization however is that changes you might make to the +configuration scripts will not be taken into account by default on the next +incremental build. A forced reconfiguration can also be required in some cases +where build scripts automatically detect sources in it’s configuration phase, +so newly added sources you add might be ignored.

+

In order to force the configuration step to be called again on the next build, +you can use bst workspace reset –soft, like so:

+

In these cases, you can perform a hard reset on the workspace using +bst workspace reset, like so:

+ +
+user@host:~/developing$ bst workspace reset --soft hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888175.340515    1925 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] INFO    Reset workspace state for hello.bst at: /home/user/buildstream/doc/examples/developing/workspace_hello
+
+

This will ensure that the next time you run the build, BuildStream will forcefully +rerun the Element.prepare() method +and cause the configuration step to occur again.

+
+
+

1.4. Closing your workspace

+

If we want to close the workspace and come back to our changes later, we can

+ +
+user@host:~/developing$ bst workspace close hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888175.754866    1953 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] INFO    Closed workspace for hello.bst
+
+

We can then reopen the workspace later using:

+ +
+user@host:~/developing$ bst workspace open --no-checkout --directory workspace_hello hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888176.179757    1981 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] STATUS  Creating workspace for element hello.bst
+[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element hello.bst
+
+

The –no-checkout option tells BuildStream not to check the source out but to +instead hard-link to the workspace_hello directory.

+

Alternatively, if we wish to discard the changes we can use

+ +
+user@host:~/developing$ bst workspace reset hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888176.598029    2009 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
+[00:00:00][        ][    main:core activity                 ] SUCCESS Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
+[--:--:--][        ][    main:core activity                 ] INFO    Closed workspace for hello.bst
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][        ][    main:core activity                 ] STATUS  Creating workspace for element hello.bst
+[--:--:--][a6a39926][    main:hello.bst                     ] START   Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
+[--:--:--][        ][    main:hello.bst                     ] START   Staging local files into CAS
+[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files into CAS
+[00:00:00][a6a39926][    main:hello.bst                     ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
+[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element hello.bst
+
+

This resets the workspace to its original state.

+

To discard the workspace completely we can do:

+ +
+user@host:~/developing$ bst workspace close --remove-dir hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888177.023013    2037 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
+[00:00:00][        ][    main:core activity                 ] SUCCESS Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
+[--:--:--][        ][    main:core activity                 ] INFO    Closed workspace for hello.bst
+
+

This will close the workspace and completely remove the workspace_hello +directory.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/docs.tgz b/docs.tgz new file mode 100644 index 000000000..99a02eb79 Binary files /dev/null and b/docs.tgz differ diff --git a/elements/compose.html b/elements/compose.html new file mode 100644 index 000000000..2a5a4e2f2 --- /dev/null +++ b/elements/compose.html @@ -0,0 +1,194 @@ + + + + + + + + + compose - Compose the output of multiple elements — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

compose - Compose the output of multiple elements

+

This element creates a selective composition of its dependencies.

+

This is normally used at near the end of a pipeline to prepare +something for later deployment.

+

Since this element’s output includes its dependencies, it may only +depend on elements as build type dependencies.

+
+
The default configuration and possible options are as such:
#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# Compose element configuration
+config:
+
+  # Whether to run the integration commands for the
+  # staged dependencies.
+  #
+  integrate: True
+
+  # A list of domains to include from each artifact, as
+  # they were defined in the element's 'split-rules'.
+  #
+  # Since domains can be added, it is not an error to
+  # specify domains which may not exist for all of the
+  # elements in this composition.
+  #
+  # The default empty list indicates that all domains
+  # from each dependency should be included.
+  #
+  include: []
+
+  # A list of domains to exclude from each artifact, as
+  # they were defined in the element's 'split-rules'.
+  #
+  # In the case that a file is spoken for by a domain
+  # in the 'include' list and another in the 'exclude'
+  # list, then the file will be excluded.
+  exclude: []
+
+  # Whether to include orphan files which are not
+  # included by any of the 'split-rules' present on
+  # a given element.
+  #
+  include-orphans: True
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/elements/filter.html b/elements/filter.html new file mode 100644 index 000000000..6bceb9052 --- /dev/null +++ b/elements/filter.html @@ -0,0 +1,281 @@ + + + + + + + + + filter - Extract a subset of files from another element — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

filter - Extract a subset of files from another element

+

Filter another element by producing an output that is a subset of +the parent element’s output. Subsets are defined by the parent element’s +split rules.

+
+

Overview

+

A filter element must have exactly one build dependency, where said +dependency is the ‘parent’ element which we would like to filter. +Runtime dependencies may also be specified, which can be useful to propagate +forward from this filter element onto its reverse dependencies. +See Dependencies to see how we specify dependencies.

+

When workspaces are opened, closed or reset on a filter element, or this +element is tracked, the filter element will transparently pass on the command +to its parent element (the sole build-dependency).

+

Example

+

Consider a simple import element, import.bst which imports the local files +‘foo’, ‘bar’ and ‘baz’ (each stored in files/, relative to the project’s root):

+
kind: import
+
+# Specify sources to import
+sources:
+- kind: local
+  path: files
+
+# Specify public domain data, visible to other elements
+public:
+  bst:
+    split-rules:
+      foo:
+      - /foo
+      bar:
+      - /bar
+
+
+
+

Note

+

We can make an element’s metadata visible to all reverse dependencies by making use +of the public: field. See the public data documentation +for more information.

+
+

In this example, import.bst will serve as the ‘parent’ of the filter element, thus +its output will be filtered. It is important to understand that the artifact of the +above element will contain the files: ‘foo’, ‘bar’ and ‘baz’.

+

Now, to produce an element whose artifact contains the file ‘foo’, and exlusively ‘foo’, +we can define the following filter, filter-foo.bst:

+
kind: filter
+
+# Declare the sole build-dependency of the filter element
+build-depends:
+- import.bst
+
+# Declare a list of domains to include in the filter's artifact
+config:
+  include:
+  - foo
+
+
+

It should be noted that an ‘empty’ include: list would, by default, include all +split-rules specified in the parent element, which, in this example, would be the +files ‘foo’ and ‘bar’ (the file ‘baz’ was not covered by any split rules).

+

Equally, we can use the exclude: statement to create the same artifact (which +only contains the file ‘foo’) by declaring the following element, exclude-bar.bst:

+
kind: filter
+
+# Declare the sole build-dependency of the filter element
+build-depends:
+- import.bst
+
+# Declare a list of domains to exclude in the filter's artifact
+config:
+  exclude:
+  - bar
+
+
+

In addition to the include: and exclude: fields, there exists an include-orphans: +(Boolean) field, which defaults to False. This will determine whether to include files +which are not present in the ‘split-rules’. For example, if we wanted to filter out all files +which are not included as split rules we can define the following element, filter-misc.bst:

+
kind: filter
+
+# Declare the sole build-dependency of the filter element
+build-depends:
+- import.bst
+
+# Filter out all files which are not declared as split rules
+config:
+  exclude:
+  - foo
+  - bar
+  include-orphans: True
+
+
+

The artifact of filter-misc.bst will only contain the file ‘baz’.

+

Below is more information regarding the the default configurations and possible options +of the filter element:

+
#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# Filter element configuration
+config:
+
+  # A list of domains to include in each artifact, as
+  # they were defined as public data in the parent
+  # element's 'split-rules'.
+  #
+  # If a domain is specified that does not exist, the
+  # filter element will fail to build.
+  #
+  # The default empty list indicates that all domains
+  # of the parent's artifact should be included.
+  #
+  include: []
+
+  # A list of domains to exclude from each artifact, as
+  # they were defined in the parent element's 'split-rules'.
+  #
+  # In the case that a file is spoken for by a domain
+  # in the 'include' list and another in the 'exclude'
+  # list, then the file will be excluded.
+  exclude: []
+
+  # Whether to include orphan files which are not
+  # included by any of the 'split-rules' present in
+  # the parent element.
+  #
+  include-orphans: False
+
+  # Whether to pass the 'integration-commands' of the
+  # parent element through the filter.
+  #
+  pass-integration: False
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/elements/import.html b/elements/import.html new file mode 100644 index 000000000..de6247d51 --- /dev/null +++ b/elements/import.html @@ -0,0 +1,174 @@ + + + + + + + + + import - Import sources directly — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

import - Import sources directly

+

Import elements produce artifacts directly from its sources +without any kind of processing. These are typically used to +import an SDK to build on top of or to overlay your build with +some configuration data.

+
+
The empty configuration is as such:
#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# The import element simply stages the given sources
+# directly to the root of the sandbox and then collects
+# the output to create an output artifact.
+#
+config:
+
+  # By default we collect everything staged, specify a
+  # directory here to output only a subset of the staged
+  # input sources.
+  source: /
+
+  # Prefix the output with an optional directory, by default
+  # the input is found at the root of the produced artifact.
+  target: /
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/elements/junction.html b/elements/junction.html new file mode 100644 index 000000000..8b5ca7186 --- /dev/null +++ b/elements/junction.html @@ -0,0 +1,405 @@ + + + + + + + + + junction - Integrate subprojects — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

junction - Integrate subprojects

+

This element acts as a window into another BuildStream project. It allows integration +of multiple projects into a single pipeline.

+
+

Overview

+
kind: junction
+
+# Specify the BuildStream project source
+sources:
+- kind: git
+  url: upstream:projectname.git
+  track: master
+  ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6
+
+# Specify the junction configuration
+config:
+
+  # Override project options
+  options:
+    machine_arch: "%{machine_arch}"
+    debug: True
+
+  # Optionally look in a subpath of the source repository for the project
+  path: projects/hello
+
+  # Optionally override elements in subprojects, including junctions.
+  #
+  overrides:
+    subproject-junction.bst: local-junction.bst
+
+  # Optionally override aliases in subprojects, to allow using mirrors
+  # defined in the parent project.
+  aliases:
+    subproject-alias: local-alias
+
+  # A default mapping can be set (defaults to none)
+  map-aliases: identity
+
+
+

With a junction element in place, local elements can depend on elements in +the other BuildStream project using element paths. +For example, if you have a toolchain.bst junction element referring to +a project which contains a gcc.bst element, you can express a build +dependency to the compiler like this:

+
build-depends:
+- junction: toolchain.bst:gcc.bst
+
+
+
+

Important

+

Limitations

+

Junction elements are only connectors which bring multiple projects together, +and as such they are not in the element dependency graph. This means that it is +illegal to depend on a junction, and it is also illegal for a junction to have +dependencies.

+

While junctions are elements, a limited set of element operations are +supported. Junction elements can be tracked and fetched like other +elements but they do not produce any artifacts, which means that they +cannot be built or staged.

+

Note that when running bst source track +on your project, elements found in subprojects are not tracked by default. +You may specify --cross-junctions to the +bst source track command to explicitly track +elements across junction boundaries.

+
+
+
+

Sources

+

The sources of a junction element define how to obtain the BuildStream project +that the junction connects to.

+

Most commands, such as bst build, will automatically +try to fetch the junction elements required to access any subproject elements which +are specified as dependencies of the targets provided.

+

Some commands, such as bst show, do not do this, and in +such cases they can be fetched explicitly using +bst source fetch:

+
bst source fetch junction.bst
+
+
+
+
+

Options

+

Junction elements can configure the project options +in the subproject, using the options configuration.

+
kind: junction
+
+...
+
+config:
+
+  # Specify the options for this subproject
+  #
+  options:
+    machine_arch: "%{machine_arch}"
+    debug: True
+
+
+

Options are never implicitly propagated across junctions, however +variables can be used to explicitly assign +configuration in a subproject which matches the toplevel project’s +configuration.

+
+
+

Overriding elements

+

It is possible to override elements in subprojects. This can be useful if for +example, you need to work with a custom variant or fork of some software in the +subproject. This is a better strategy than overlapping and overwriting shared +libraries built by the subproject later on, as we can ensure that reverse dependencies +in the subproject are built against the overridden element.

+

Overridding elements allows you to build on top of an existing project +and benefit from updates and releases for the vast majority of the upstream project, +even when there are some parts of the upstream project which need to be customized +for your own applications.

+

Even junction elements in subprojects can be overridden, this is sometimes important +in order to reconcile conflicts when multiple projects depend on the same subproject, +as discussed below.

+
kind: junction
+
+...
+
+config:
+
+  # Override elements in a junctioned project
+  #
+  overrides:
+    subproject-element.bst: local-element.bst
+
+
+

It is also possible to override elements in deeply nested subprojects, using +project relative junction paths:

+
kind: junction
+
+...
+
+config:
+
+  # Override deeply nested elements
+  #
+  overrides:
+    subproject.bst:subsubproject-element.bst: local-element.bst
+
+
+
+

Attention

+

Overriding an element causes your project to completely define the +element being overridden, which means you will no longer receive updates +or security patches to the element in question when updating to newer +versions and releases of the upstream project.

+

As such, overriding elements is only recommended in cases where the +element is very significantly redefined.

+

Such cases include cases when you need a newer version of the element than +the one maintained by the upstream project you are using as a subproject, +or when you have significanly modified the code in your own custom ways.

+

If you only need to introduce a security patch, then it is recommended that +you create your own downstream branch of the upstream project, not only will +this allow you to more easily consume updates with VCS tools like git rebase, +but it will also be more convenient for submitting your security patches +to the upstream project so that you can drop them in a future update.

+

Similarly, if you only need to enable/disable a specific feature of a module, +it is also preferrable to use a downstream branch of the upstream project. +In such a case, it is also worth trying to convince the upstream project to +support a project option for your specific element +configuration, if it would be of use to other users too.

+
+
+
+

Nested Junctions

+

Junctions can be nested. That is, subprojects are allowed to have junctions on +their own. Nested junctions in different subprojects may point to the same +project, however, in most use cases the same project should be loaded only once.

+

As the junctions may differ in source version and options, BuildStream cannot +simply use one junction and ignore the others. Due to this, BuildStream requires +the user to resolve conflicting nested junctions, and will provide an error +message whenever a conflict is detected.

+
+

Overriding subproject junctions

+

If your project and a subproject share a subproject in common, then one way +to resolve the conflict is to override the subproject’s junction with a local +in your project.

+

You can override junctions in a subproject in the junction declaration +of that subproject, e.g.:

+
kind: junction
+
+# Here we are junctioning "subproject" which
+# also junctions "subsubproject", which we also
+# use directly.
+#
+sources:
+- kind: git
+  url: https://example.com/subproject.git
+
+config:
+  # Override `subsubproject.bst` in the subproject using
+  # the locally declared `local-subsubproject.bst` junction.
+  #
+  overrides:
+    subsubproject.bst: local-subsubproject.bst
+
+
+

When declaring the overrides dictionary, the keys (on the left side) +refer to junction paths which are relative +to the subproject you are declaring. The values (on the right side) refer +to junction paths which are relative to the +project in which your junction is declared.

+
+

Warning

+

This approach modifies your subproject, causing its output artifacts +to differ from that project’s expectations.

+

If you rely on validation and guarantees provided by the organization +which maintains the subproject, then it is desirable to avoid overriding +any details from that upstream project.

+
+
+
+

Linking to other junctions

+

Another way to resolve the conflict when your project and a subproject both +junction a common project, is to simply reuse the same junction from the +subproject in your toplevel project.

+

This is preferable to overrides because you can avoid modifying the +subproject you would otherwise be changing with an override.

+

A convenient way to reuse a nested junction in a higher level project +is to create a link element to that subproject’s +junction. This will help you avoid redundantly typing out longer +element paths in your project’s +dependency declarations.

+

This way you can simply create the link once +in your project and use it locally to depend on elements in a nested +subproject.

+

Example:

+
# Declare the `subsubproject-link.bst` link element, which
+# is a symbolic link to the junction declared in the subproject
+#
+kind: link
+
+config:
+  target: subproject.bst:subsubproject.bst
+
+
+
# Depend on elements in the subsubproject using
+# the subproject's junction directly
+#
+kind: autotools
+
+depends:
+- subsubproject-link.bst:glibc.bst
+
+
+
+

Tip

+

When reconciling conflicting junction declarations to the +same subproject, it is also possible to use a locally defined +link element from one subproject to +override another junction to the same project in an adjacent +subproject.

+
+
+
+

Multiple project instances

+

By default, loading the same project more than once will result +in a conflicting junction error. There are some use cases which +demand that you load the same project more than once in the same +build pipeline.

+

In order to allow the loading of multiple instances of the same project +in the same build pipeline, please refer to the +relevant project.conf documentation.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/elements/link.html b/elements/link.html new file mode 100644 index 000000000..5b59e67a0 --- /dev/null +++ b/elements/link.html @@ -0,0 +1,155 @@ + + + + + + + + + link - Link elements — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ + + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/elements/manual.html b/elements/manual.html new file mode 100644 index 000000000..ecd63f052 --- /dev/null +++ b/elements/manual.html @@ -0,0 +1,182 @@ + + + + + + + + + manual - Manual build element — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

manual - Manual build element

+

The most basic build element does nothing but allows users to +add custom build commands to the array understood by the BuildElement

+
+
The empty configuration is as such:
#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# Manual build element does not provide any default
+# build commands
+config:
+
+  # Commands for configuring the software
+  #
+  configure-commands: []
+
+  # Commands for building the software
+  #
+  build-commands: []
+
+  # Commands for installing the software into a
+  # destination folder
+  #
+  install-commands: []
+
+  # Commands for stripping installed binaries
+  #
+  strip-commands:
+  - |
+    %{strip-binaries}
+
+
+
+
+

See built-in functionality documentation for +details on common configuration options for build elements.

+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/elements/script.html b/elements/script.html new file mode 100644 index 000000000..b81429bbf --- /dev/null +++ b/elements/script.html @@ -0,0 +1,196 @@ + + + + + + + + + script - Run scripts to create output — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

script - Run scripts to create output

+

This element allows one to run some commands to mutate the +input and create some output.

+
+

Note

+

Script elements may only specify build dependencies. See +the format documentation for more +detail on specifying dependencies.

+
+
+
The default configuration and possible options are as such:
#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# The script element allows staging elements into specific locations
+# via it's "location" dependency configuration
+#
+# For example, if you want to stage "foo-tools.bst" into the "/" of
+# the sandbox at buildtime, and the "foo-system.bst" element into
+# the %{build-root}, you can do so as follows:
+#
+# build-depends:
+# - foo-tools.bst
+# - filename: foo-system.bst
+#   config:
+#     location: "%{build-root}"
+#
+# Note: the default of the "location" parameter is "/", so it is not
+#       necessary to specify the location if you want to stage the
+#       element in "/"
+#
+
+# Common script element variables
+variables:
+  # Defines the directory commands will be run from.
+  cwd: /
+
+# Script element configuration
+config:
+
+  # Defines whether to run the sandbox with '/' read-only.
+  # It is recommended to set root as read-only wherever possible.
+  root-read-only: False
+
+  # List of commands to run in the sandbox.
+  commands: []
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/elements/stack.html b/elements/stack.html new file mode 100644 index 000000000..f9ccf082f --- /dev/null +++ b/elements/stack.html @@ -0,0 +1,205 @@ + + + + + + + + + stack - Symbolic Element for dependency grouping — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

stack - Symbolic Element for dependency grouping

+

Stack elements are simply a symbolic element used for representing +a logical group of elements.

+

All dependencies declared in stack elements must always be both +build and runtime dependencies.

+

Example:

+
kind: stack
+
+# Declare all of your dependencies in the `depends` list.
+depends:
+- libc.bst
+- coreutils.bst
+
+
+
+

Note

+

Unlike other elements, whose cache keys are a unique identifier +of the contents of the artifacts they produce, stack elements do +not produce any artifact content. Instead, the cache key of an artifact +is a unique identifier for the assembly of its own dependencies.

+
+
+

Using intermediate stacks

+

Using a stack element at intermediate levels of your build graph +allows you to abstract away some parts of your project into logical +subsystems which elements can more conveniently depend on as a whole.

+

In addition to the added convenience, it will allow you to more +easily change the implementation of a subsystem later on, without needing +to update many reverse dependencies to depend on new elements, or even +allow you to conditionally implement a subsystem with various implementations +depending on what project options were specified at +build time.

+
+
+

Using toplevel stacks

+

Stack elements can also be useful as toplevel targets in your build graph +to simply indicate all of the components which need to be built for a given +system to be complete, or for your integration pipeline to be successful.

+
+

Checking out and deploying toplevel stacks

+

In case that your software is built remotely, it is possible to checkout +the built content of a stack on your own machine for the purposes of +inspection or further deployment.

+

To accomplish this, you will need to know the cache key of the stack element +which was built remotely, possibly by inspecting the remote build log or by +deriving it with an equally configured BuildStream project, and you will +need read access to the artifact cache server which the build was uploaded to, +this should be configured in your user configuration file.

+

You can then checkout the remotely built stack using the +bst artifact checkout command and providing +it with the artifact name:

+

Example:

+
bst artifact checkout --deps build --pull --integrate \
+    --directory `pwd`/checkout \
+    project/stack/788da21e7c1b5818b7e7b60f7eb75841057ff7e45d362cc223336c606fe47f27
+
+
+
+

Note

+

It is possible to checkout other elements in the same way, however stack +elements are uniquely suited to this purpose, as they cannot have +runtime only dependencies, and consequently +their cache keys are always a unique representation of their collective +dependencies.

+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/examples/git-mirror.html b/examples/git-mirror.html new file mode 100644 index 000000000..e8aa6cd0f --- /dev/null +++ b/examples/git-mirror.html @@ -0,0 +1,256 @@ + + + + + + + + + Creating and using a git mirror — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Creating and using a git mirror

+

This is an example of how to create a git mirror using git’s +git-http-backend and +lighttpd.

+
+

Prerequisites

+

You will need git installed, and git-http-backend must be present. It is assumed +that the git-http-backend binary exists at /usr/lib/git-core/git-http-backend.

+

You will need lighttpd installed, and at the bare minimum has the modules +mod_alias, mod_cgi, and mod_setenv.

+

I will be using gnome-modulesets as an example, which can be cloned from +http://gnome7.codethink.co.uk/gnome-modulesets.git.

+
+
+

Starting a git http server

+
+

1. Set up a directory containing mirrors

+

Choose a suitable directory to hold your mirrors, e.g. /var/www/git.

+

Place the git repositories you want to use as mirrors in the mirror dir, e.g. +git clone --mirror http://git.gnome.org/browse/yelp-xsl /var/www/git/yelp-xsl.git.

+
+
+

2. Configure lighttpd

+

Write out a lighttpd.conf as follows:

+
server.document-root = "/var/www/git/"
+server.port = 3000
+server.modules = (
+     "mod_alias",
+     "mod_cgi",
+     "mod_setenv",
+)
+
+alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
+$HTTP["url"] =~ "^/git" {
+     cgi.assign = ("" => "")
+     setenv.add-environment = (
+             "GIT_PROJECT_ROOT" => "/var/www/git",
+             "GIT_HTTP_EXPORT_ALL" => ""
+     )
+}
+
+
+
+

Note

+

If you have your mirrors in another directory, replace /var/www/git/ with that directory.

+
+
+
+

3. Start lighttpd

+

lighttpd can be invoked with the command-line lighttpd -D -f lighttpd.conf.

+
+
+

4. Test that you can fetch from it

+

We can then clone the mirrored repo using git via http with +git clone http://127.0.0.1:3000/git/yelp-xsl.

+
+

Note

+

If you have set server.port to something other than the default, you will +need to replace the ‘3000’ in the command-line.

+
+
+
+

5. Configure the project to use the mirror

+

To add this local http server as a mirror, add the following to the project.conf:

+
mirrors:
+- name: local-mirror
+  aliases:
+    git_gnome_org:
+    - http://127.0.0.1:3000/git/
+
+
+
+
+

6. Test that the mirror works

+

We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.

+
aliases:
+  git_gnome_org: https://www.example.com/invalid/url/
+
+
+

Now, if you build an element that uses the source you placed in the mirror +(e.g. bst build core-deps/yelp-xsl.bst), you will see that it uses your mirror.

+
+
+
+

Bonus: lighttpd conf for git and tar

+

For those who have also used the tar-mirror tutorial, +a combined lighttpd.conf is below:

+
server.document-root = "/var/www/"
+server.port = 3000
+server.modules = (
+        "mod_alias",
+        "mod_cgi",
+        "mod_setenv",
+)
+
+alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
+$HTTP["url"] =~ "^/git" {
+        cgi.assign = ("" => "")
+        setenv.add-environment = (
+                "GIT_PROJECT_ROOT" => "/var/www/git",
+                "GIT_HTTP_EXPORT_ALL" => ""
+        )
+} else $HTTP["url"] =~ "^/tar" {
+        dir-listing.activate = "enable"
+}
+
+
+
+
+

Further reading

+

If this mirror isn’t being used exclusively in a secure network, it is strongly +recommended you use SSL.

+

This is the bare minimum required to set up a git mirror. A large, public project +would prefer to set it up using the +git protocol, +and a security-conscious project would be configured to use +git over SSH.

+

Lighttpd is documented on its wiki.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/examples/tar-mirror.html b/examples/tar-mirror.html new file mode 100644 index 000000000..9bb420edb --- /dev/null +++ b/examples/tar-mirror.html @@ -0,0 +1,216 @@ + + + + + + + + + Creating and using a tar mirror — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Creating and using a tar mirror

+

This is an example of how to create a tar mirror using +lighttpd.

+
+

Prerequisites

+

You will need lighttpd installed.

+

I will be using gnome-modulesets as an example, which can be cloned from +http://gnome7.codethink.co.uk/gnome-modulesets.git.

+
+
+

Starting a tar server

+
+

1. Set up a directory containing mirrors

+

Choose a suitable directory to hold your mirrored tar files, e.g. /var/www/tar.

+

Place the tar files you want to use as mirrors in your mirror dir, e.g.

+
mkdir -p /var/www/tar/gettext
+wget -O /var/www/tar/gettext/gettext-0.19.8.1.tar.xz https://ftp.gnu.org/gnu/gettext/gettext-0.19.8.1.tar.xz
+
+
+
+
+

2. Configure lighttpd

+

Write out a lighttpd.conf as follows:

+
server.document-root = "/var/www/tar/"
+server.port = 3000
+
+dir-listing.activate = "enable"
+
+
+
+

Note

+

If you have your mirrors in another directory, replace /var/www/tar/ with that directory.

+
+
+

Note

+

An example lighttpd.conf that works for both git and tar services is available +here

+
+
+
+

3. Start lighttpd

+

lighttpd can be invoked with the command-line lighttpd -D -f lighttpd.conf.

+
+
+

4. Test that you can fetch from it

+

We can then download the mirrored file with wget 127.0.0.1:3000/tar/gettext/gettext-0.19.8.1.tar.xz.

+
+

Note

+

If you have set server.port to something other than the default, you will need +to replace the ‘3000’ in the command-line.

+
+
+
+

5. Configure the project to use the mirror

+

To add this local http server as a mirror, add the following to the project.conf:

+
mirrors:
+- name: local-mirror
+  aliases:
+    ftp_gnu_org:
+    - http://127.0.0.1:3000/tar/
+
+
+
+
+

6. Test that the mirror works

+

We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.

+
aliases:
+  ftp_gnu_org: https://www.example.com/invalid/url/
+
+
+

Now, if you build an element that uses the source you placed in the mirror +(e.g. bst build core-deps/gettext.bst), you will see that it uses your mirror.

+
+
+
+

Further reading

+

If this mirror isn’t being used exclusively in a secure network, it is strongly +recommended you use SSL.

+

Lighttpd is documented on its wiki.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/format_declaring.html b/format_declaring.html new file mode 100644 index 000000000..3a4fb8189 --- /dev/null +++ b/format_declaring.html @@ -0,0 +1,715 @@ + + + + + + + + + Declaring elements — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Declaring elements

+
+

Element basics

+

Here is a rather complete example using the autotools element kind and git source kind:

+
# Specify the kind of element this is
+kind: autotools
+
+# Specify some dependencies
+depends:
+- element1.bst
+- element2.bst
+
+# Specify the source which should be built
+sources:
+- kind: git
+  url: upstream:modulename.git
+  track: master
+  ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6
+
+# Override some variables
+variables:
+  sysconfdir: "%{prefix}/etc"
+
+# Tweak the sandbox shell environment
+environment:
+  LD_LIBRARY_PATH: /some/custom/path
+
+# Specify the configuration of the element
+config:
+
+  # Override autotools element default configure-commands
+  configure-commands:
+  - "%{configure} --enable-fancy-feature"
+
+# Specify public domain data, visible to other elements.
+public:
+  bst:
+    integration-commands:
+    - /usr/bin/update-fancy-feature-cache
+
+# Specify a user id and group id to use in the build sandbox.
+sandbox:
+  build-uid: 0
+  build-gid: 0
+
+
+

For most use cases you would not need to specify this much detail, we’ve provided +details here in order to have a more complete initial example.

+

Let’s break down the above and give a brief explanation of what these attributes mean.

+
+

Element names and paths

+

An element name is the filename of an element relative to the project’s +element path.

+

Element names are the identifiers used to refer to elements, they are used +to specify an element’s dependencies, to select +elements to build on the command line, and they are arbitrarily +used in various element specific configuration surfaces, for example the +target configuration of the link element is also +an element name.

+
+

Addressing elements

+

When addressing elements in a single project, it is sufficient to use +the element name as a dependency or configuration parameter.

+

When muliple projects are connected through junction +elements, there is a need to address elements which are not in the same +project but in a junctioned subproject. In the case that you need to +address elements across junction boundaries, one must use element paths.

+

An element path is a path to the element indicating the junction +elements leading up to the project, separated by : symbols, e.g.: +junction.bst:element.bst.

+

Elements can be address across multiple junction boundaries with multiple +: separators, e.g.: junction.bst:junction.bst:element.bst.

+
+
+

Element naming rules

+

When naming the elements, use the following rules:

+
    +

  • The name of the file must have the .bst extension.

    • +

  • All characters in the name must be printable 7-bit ASCII characters.

    • +

  • Following characters are reserved and must not be part of the name:

    +
      +

    • < (less than)

      • +

    • > (greater than)

      • +

    • : (colon)

      • +

    • " (double quote)

      • +

    • / (forward slash)

      • +

    • \ (backslash)

      • +

    • | (vertical bar)

      • +

    • ? (question mark)

      • +

    • * (asterisk)

      • +
    +
    • +
+
+
+
+

Kind

+
# Specify the kind of element this is
+kind: autotools
+
+
+

The kind attribute specifies which plugin will be operating on the element’s input to +produce its output. Plugins define element types and each of them can be referred to by +name with the kind attribute.

+

To refer to a third party plugin, prefix the plugin with its package, for example:

+
kind: buildstream-plugins:dpkg_build
+
+
+
+
+

Depends

+
# Specify some dependencies
+depends:
+- element1.bst
+- element2.bst
+
+
+

Relationships between elements are specified with the depends attribute. Elements +may depend on other elements by specifying the element names +they depend on here.

+

See Dependencies for more information on the dependency model.

+
+
+

Build-Depends

+
# Specify some build-dependencies
+build-depends:
+- element1.bst
+- element2.bst
+
+
+

Build dependencies between elements can be specified with the build-depends attribute. +The above code snippet is equivalent to:

+
# Specify some build-dependencies
+depends:
+- filename: element1.bst
+  type: build
+- filename: element2.bst
+  type: build
+
+
+

See Dependencies for more information on the dependency model.

+
+
+

Runtime-Depends

+
# Specify some runtime-dependencies
+runtime-depends:
+- element1.bst
+- element2.bst
+
+
+

Runtime dependencies between elements can be specified with the runtime-depends attribute. +The above code snippet is equivalent to:

+
# Specify some runtime-dependencies
+depends:
+- filename: element1.bst
+  type: runtime
+- filename: element2.bst
+  type: runtime
+
+
+

See Dependencies for more information on the dependency model.

+
+
+

Sources

+
# Specify the source which should be built
+sources:
+- kind: git
+  url: upstream:modulename.git
+  track: master
+  ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6
+
+
+

Here we specify some input for the element, any number of sources may be specified. +By default the sources will be staged in the root of the element’s build directory +in the build sandbox, but sources may specify a directory attribute to control +where the sources will be staged. The directory attribute may specify a build +sandbox relative subdirectory.

+

For example, one might encounter a component which requires a separate data package +in order to build itself, in this case the sources might be listed as:

+
sources:
+
+# Specify the source which should be built
+- kind: git
+  url: upstream:modulename.git
+  track: master
+  ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6
+
+# Specify the data package we need for build frobnication,
+# we need it to be unpacked in a src/frobdir
+- kind: tarball
+  directory: src/frobdir
+  url: data:frobs.tgz
+  ref: 9d4b1147f8cf244b0002ba74bfb0b8dfb3...
+
+
+

Like Elements, Source types are plugins which are indicated by the kind attribute. +Asides from the common kind and directory attributes which may be applied to all +Sources, refer to the Source specific documentation for meaningful attributes for the +particular Source.

+
+
+

Variables

+
# Override some variables
+variables:
+  sysconfdir: "%{prefix}/etc"
+
+
+

Variables can be declared or overridden from an element. Variables can also be +declared and overridden in the Project configuration

+

See Using variables below for a more in depth discussion on variables in BuildStream.

+
+
+

Environment

+
# Tweak the sandbox shell environment
+environment:
+  LD_LIBRARY_PATH: /some/custom/path
+
+
+

Environment variables can be set to literal values here, these environment +variables will be effective in the Sandbox where +build instructions are run for this element.

+

Environment variables can also be declared and overridden in the Project configuration

+
+
+

Config

+
# Specify the configuration of the element
+config:
+
+  # Override autotools element default configure-commands
+  configure-commands:
+  - "%{configure} --enable-fancy-feature"
+
+
+

Here we configure the element itself. The autotools element provides sane defaults for +building sources which use autotools. Element default configurations can be overridden +in the project.conf file and additionally overridden in the declaration of an element.

+

For meaningful documentation on what can be specified in the config section for a given +element kind, refer to the element specific documentation.

+
+
+

Public

+
# Specify public domain data, visible to other elements.
+public:
+  bst:
+    integration-commands:
+    - /usr/bin/update-fancy-feature-cache
+
+
+

Metadata declared in the public section of an element is visible to +any other element which depends on the declaring element in a given pipeline. +BuildStream itself consumes public data from the bst domain. The integration-commands +demonstrated above for example, describe commands which should be run in an +environment where the given element is installed but before anything should be run.

+

An element is allowed to read domain data from any element it depends on, and users +may specify additional domains to be understood and processed by their own element +plugins.

+

The public data keys which are recognized under the bst domain +can be viewed in detail in the builtin public data section.

+
+
+

Sandbox

+

Configuration for the build sandbox (other than environment variables) +can be placed in the sandbox configuration. The UID and GID used by the user +in the group can be specified, as well as the desired OS and machine +architecture. Possible machine architecture follow the same list as specified in +the architecture option.

+
# Specify a user id and group id to use in the build sandbox.
+sandbox:
+  build-uid: 1003
+  build-gid: 1001
+
+
+

BuildStream normally uses uid 0 and gid 0 (root) to perform all +builds. However, the behaviour of certain tools depends on user id, +behaving differently when run as non-root. To support those builds, +you can supply a different uid or gid for the sandbox. Only +bwrap-style sandboxes support custom user IDs at the moment, and hence +this will only work on Linux host platforms.

+
# Specify build OS and architecture
+sandbox:
+  build-os: AIX
+  build-arch: power-isa-be
+
+
+

When building locally, if these don’t match the host machine then generally the +build will fail. The exception is when the OS is Linux and the architecture +specifies an x86-32 build on an x86-64 machine, or aarch32 build on +a aarch64 machine, in which case the linux32 command is prepended to the +bubblewrap command.

+

When building remotely, the OS and architecture are added to the Platform +field in the Command uploaded. Whether this actually results in a building +the element for the desired OS and architecture is dependent on the server +having implemented these options the same as buildstream.

+
+
+
+

Dependencies

+

The dependency model in BuildStream is simplified by treating software distribution +and software building as separate problem spaces. This is to say that one element +can only ever depend on another element but never on a subset of the product which +another element produces.

+

In this section we’ll quickly go over the few features BuildStream offers in its +dependency model.

+
+

Expressing dependencies

+

Dependencies in BuildStream are parameterizable objects, however as demonstrated +in the above example, they can also be expressed as simple +strings as a convenience shorthand in most cases, whenever the default dependency +attributes are suitable.

+
+

Note

+

Note the order in which element dependencies are declared in the depends, +build-depends and runtime-depends lists are not meaningful.

+
+

Dependency dictionary:

+
# Fully specified dependency
+depends:
+- filename: foo.bst
+  type: build
+  junction: baseproject.bst
+  strict: false
+
+
+

Attributes:

+
    +

  • filename

    +

    The element name to depend on, or a list of mutiple element names.

    +

    Specifying multiple element names in a single dependency will result in multiple dependencies +being declared with common properties.

    +

    For example, one can declare multiple build dependencies with the same junction:

    +
    # Declare three build dependencies from subproject.bst
+depends:
+- type: build
+  junction: subproject.bst
+  filename:
+  - element-a.bst
+  - element-b.bst
+  - element-c.bst
+
    +
    +
    • +

  • junction

    +

    This attribute can be used to specify the junction portion of the element name +separately from the project local element name.

    +

    This should be the element name of the junction element +in the local project, possibly followed by other junctions in subprojects leading +to the project in which the element you want to depend on resides.

    +

    In the case that a junction is specified, the filename attribute indicates an +element in the junctioned project.

    +
    • +

  • type

    +

    This attribute is used to express the dependency type. +This field is not permitted in the build-depends or +runtime-depends lists.

    +
    • +

  • strict

    +

    This attribute can be used to specify that this element should +be rebuilt when the dependency changes, even when +strict mode has been turned off.

    +

    This is appropriate whenever a dependency’s output is consumed +verbatim in the output of the depending element, for instance +when static linking is in use.

    +
    • +

  • config

    +

    This attribute defines the custom dependency configuration, +which is supported by select Element implementations.

    +

    Elements which support dependency configuration do so +by implementing the +Element.configure_dependencies() +abstract method. It is up to each element or abstract element class to +document what is supported in their dependency configuration.

    +
    +

    Attention

    +

    It is illegal to declare dependency configuration +on runtime dependencies, since runtime dependencies are not visible to the depending +element.

    +
    +
    • +
+
+
+

Redundant dependency declarations

+

It is permitted to declare dependencies multiple times on the same element in the same +element declaration, the result will be an inclusive OR of all configurations you have +expressed in the redundant dependencies on the same element.

+
    +

  • If a dependency is defined once as a build dependency and once as a runtime +dependency type, then the resulting dependency +type will be all

    • +

  • If any of the redundantly declared dependencies are specified as strict, then +the resulting dependency will be strict.

    • +
+

Declaring redundant dependencies on the same element can be interesting when you +need to specify multiple dependency configurations +for the same element. For example, one might want to stage the same dependency +in multiple locations in the build sandbox.

+
+
+

Cross-junction dependencies

+

As explained in the element name section +on element addressing, elements can be addressed across junction boundaries +using element paths such as junction.bst:element.bst. An element +at any depth can be specified by specifying multiple junction elements.

+

For example, one can specify a subproject element dependency with +the following syntax:

+
build-depends:
+- baseproject.bst:element.bst
+
+
+

And one can specify an element residing in a sub-subproject as a +dependency like so:

+
depends:
+- baseproject.bst:middleproject.bst:element.bst
+
+
+
+
+

Dependency types

+

The dependency type attribute defines what the dependency is required for +and is essential to how BuildStream plots a build plan.

+

There are three types which one can specify for a dependency:

+
    +

  • build

    +

    A build dependency type states that the given element’s product must +be staged in order to build the depending element. Depending on an element +which has build dependencies will not implicitly depend on that element’s +build dependencies.

    +

    For convenience, these can be specified under the build-depends +list.

    +
    • +

  • runtime

    +

    A runtime dependency type states that the given element’s product +must be present for the depending element to function. An element’s +runtime dependencies are not available to the element at build time.

    +

    For convenience, these can be specified under the runtime-depends +list.

    +
    • +

  • all

    +

    An all dependency is the default dependency type. If all is specified, +or if type is not specified at all, then it is assumed that the dependency +is required both at build time and runtime.

    +
    • +
+
+

Note

+

It is assumed that a dependency which is required for building an +element must run while building the depending element. This means that +build depending on a given element implies that that element’s +runtime dependencies will also be staged for the purpose of building.

+
+
+
+
+

Using variables

+

Variables in BuildStream are a way to make your build instructions and +element configurations more dynamic.

+
+

Referring to variables

+

Variables are expressed as %{...}, where ... must contain only +alphanumeric characters and the separators _ and -. Further, the +first letter of ... must be an alphabetic character.

+
This is release version %{version}
+
+
+
+
+

Declaring and overriding variables

+

To declare or override a variable, one need only specify a value +in the relevant variables section:

+
variables:
+  hello: Hello World
+
+
+

You can refer to another variable while declaring a variable:

+
variables:
+  release-text: This is release version %{version}
+
+
+

The order in which you declare variables is arbitrary, so long as there is no cyclic +dependency and that all referenced variables are declared, the following is fine:

+
variables:
+  release-text: This is release version %{version}
+  version: 5.5
+
+
+
+

Note

+

It should be noted that variable resolution only happens after all +Element Composition has already taken place.

+

This is to say that overriding %{version} at a higher priority will affect +the final result of %{release-text}.

+
+

Example:

+
kind: autotools
+
+# Declare variable, expect %{version} was already declared
+variables:
+  release-text: This is release version %{version}
+
+config:
+
+  # Customize the installation
+  install-commands:
+  - |
+    %{make-install} RELEASE_TEXT="%{release-text}"
+
+
+
+
+

Variables declared by BuildStream

+

BuildStream declares a set of builtin +variables that may be overridden. In addition, the following +read-only variables are also dynamically declared by BuildStream:

+
    +

  • element-name

    +

    The name of the element being processed (e.g base/alpine.bst).

    +
    • +

  • project-name

    +

    The name of project where BuildStream is being used.

    +
    • +

  • project-root & project-root-uri

    +

    The directory where the project is located on the host.

    +

    This variable is only available when declaring +source alias values or +source mirror values and allows +access to files in a project on the build host.

    +
      +

    • The project-root variable is a regular absolute path

      • +

    • The project-root-uri variable is a properly quoted file:// URI

      • +
    +
    +

    Tip

    +

    Use this variable to declare source alias values +to refer to files which you store as a part of your project, e.g. tarballs +which you have committed to you BuildStream project.

    +
    +
    +

    Attention

    +

    This feature has been provided for convenience when putting together a +project without the use of proper infrastructure.

    +

    A better long term solution for accessing internal binaries and source +code is to setup internal infrastructure in your organization and use +the regular ways to access these sources from a well known internal URI.

    +
    +
    • +

  • toplevel-root & toplevel-root-uri

    +

    The directory where the toplevel project is located on the host.

    +

    This variable is only available when declaring +source alias values or +source mirror values and allows +access to files in a project on the build host.

    +
      +

    • The toplevel-root variable is a regular absolute path

      • +

    • The toplevel-root-uri variable is a properly quoted file:// URI

      • +
    +
    +

    Tip

    +

    Use this variable to declare source alias values +to refer to files which you do not store as a part of your project, e.g. +tarballs or git repositories which must be placed in a directory within +the toplevel project before running the build.

    +
    +
    +

    Attention

    +

    This feature has been provided for convenience when putting together a +project without the use of proper infrastructure.

    +

    A better long term solution for accessing internal binaries and source +code is to setup internal infrastructure in your organization and use +the regular ways to access these sources from a well known internal URI.

    +
    +
    • +

  • max-jobs

    +

    Maximum number of parallel build processes within a given +build, support for this is conditional on the element type +and the build system used (any element using ‘make’ can +implement this).

    +
    • +
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/format_intro.html b/format_intro.html new file mode 100644 index 000000000..8296a4072 --- /dev/null +++ b/format_intro.html @@ -0,0 +1,395 @@ + + + + + + + + + Introduction — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Introduction

+

At the core of BuildStream is a data model of Elements which +are parsed from .bst files in a project directory and configured from a few different +sources.

+

When BuildStream loads your project, various levels of composition occur, allowing +configuration on various levels with different priority.

+

This page provides an introduction to the project directory structure, explains the +basic directives supported inherently throughout the format, and outlines how composition +occurs and what configurations are considered in which order.

+

The meaning of the various constructs expressed in the BuildStream format are covered +in other sections of the documentation.

+
+

Directory structure

+

A BuildStream project is a directory consisting of:

+
    +

  • A project configuration file

    • +

  • BuildStream element files

    • +

  • Optional user defined plugins

    • +

  • An optional project.refs file

    • +
+

A typical project structure may look like this:

+
myproject/project.conf
+myproject/project.refs
+myproject/elements/element1.bst
+myproject/elements/element2.bst
+myproject/elements/...
+myproject/plugins/customelement.py
+myproject/plugins/customelement.yaml
+myproject/plugins/...
+
+
+

Except for the project configuration file, the user is allowed to structure +their project directory in any way. For documentation on the format of the project +configuration file, refer to the Project configuration documentation.

+

Simpler projects may choose to place all element definition files at the +root of the project directory while more complex projects may decide to +put stacks in one directory and other floating elements into other directories, +perhaps placing deployment elements in another directory, this is all fine.

+

The important part to remember is that when you declare dependency relationships, +a project relative path to the element one depends on must be provided.

+
+
+

Composition

+

Below are the various sources of configuration which go into an element or source in the +order in which they are applied. Configurations which are applied later have a higher +priority and override configurations which precede them.

+
+

1. Builtin defaults

+

The builtin defaults provide a set of builtin +default default values for project.conf.

+

The project wide defaults defined in the builtin project configuration, such as the +variables or environment sections, form the base configuration of all elements.

+
+
+

2. Project configuration

+

The project wide defaults specified in your +project.conf are now applied on top of builtin defaults.

+

Defaults such as the variables or +environment which are specified in +your project.conf override the builtin defaults for elements.

+

Note that plugin type specific configuration +in project.conf is not applied until later.

+
+
+

3. Plugin defaults

+

Elements and Sources are all implemented as plugins.

+

Each Element plugin installs a .yaml file along side their plugin to +define the default variables, environment and config. The config +is element specific and as such this is the first place where defaults +can be set on the config section.

+

The variables and environment specified in the declaring plugin’s +defaults here override the project configuration defaults for the given +element kind.

+

Source plugins do not have a .yaml file, and do not have variables or +environment.

+
+
+

4. Project configuration overrides

+

The project.conf now gives you another opportunity to +override configuration on a per plugin basis.

+

Configurations specified in the elements or +sources sections of the project.conf +will override the given plugin’s defaults.

+

In this phase, it is possible to override any configurations of a given plugin, +including configuration in element specific config sections.

+

See also Overriding plugin defaults

+
+
+

5. Plugin declarations

+

Finally, after having resolved any conditionals +in the parsing phase of loading element declarations; the configurations specified in a +.bst file have the last word on any configuration in the data model.

+
+
+
+

Directives

+
+

(?) Conditionals

+

The (?) directive allows expression of conditional statements which +test project option values.

+

The (?) directive may appear as a key in any dictionary expressed +in YAML, and its value is a list of conditional expressions. Each conditional +expression must be a single key dictionary, where the key is the conditional +expression itself, and the value is a dictionary to be composited into the +parent dictionary containing the (?) directive if the expression evaluates +to a truthy value.

+

Example:

+
variables:
+  prefix: "/usr"
+  enable-debug: False
+  (?):
+  - relocate == True:
+      prefix: "/opt"
+  - debug == True:
+      enable-debug: True
+
+
+

Expressions are evaluated in the specified order, and each time an expression +evaluates to a truthy value, its value will be composited to the parent dictionary +in advance of processing other elements, allowing for logically overriding previous +decisions in the condition list.

+

Nesting of conditional statements is also supported.

+

Example:

+
variables:
+  enable-logging: False
+  enable-debug: False
+  (?):
+  - logging == True:
+      enable-logging: True
+      (?):
+      - debugging == True:
+          enable-debug: True
+
+
+

Conditionals are expressed in a pythonic syntax, the specifics for +testing the individually supported option types are described in +their respective documentation.

+

Compound conditionals are also allowed.

+

Example:

+
variables:
+  enable-debug: False
+  (?):
+  - (logging == True and debugging == True):
+      enable-debug: True
+
+
+
+

Important

+

Conditional statements are guaranteed to always be resolved in the +context of the project where the conditional statement is declared.

+

When including a file from a +subproject, any conditionals expressed in that file will already be +resolved in the context of the subproject which the file was included +from.

+
+
+
+

(!) Assertions

+

Assertions allow the project author to abort processing and present +a custom error message to the user building their project.

+

This is only useful when used with conditionals, allowing the project +author to assert some invalid configurations.

+

Example:

+
variables:
+  (?):
+  - (logging == False and debugging == True):
+
+      (!): |
+
+        Impossible to print any debugging information when
+        logging is disabled.
+
+
+
+
+

(<) List Prepend

+

Indicates that the list should be prepended to the target list, +instead of the default behavior which is to replace the target list.

+

Example:

+
config:
+  configure-commands:
+    # Before configuring, lets make sure we're using
+    # the latest config.sub & config.guess
+    (<):
+    - cp %{datadir}/automake-*/config.{sub,guess} .
+
+
+
+
+

(>) List Append

+

Indicates that the list should be appended to the target list, instead +of the default behavior which is to replace the target list.

+

Example:

+
public:
+  bst:
+    split-rules:
+      devel:
+        # This element also adds some extra stubs which
+        # need to be included in the devel domain
+        (>):
+        - "%{libdir}/*.stub"
+
+
+
+
+

(=) List Overwrite

+

Indicates that the list should be overwritten completely.

+

This exists mostly for completeness, and we recommend using literal +lists most of the time instead of list overwrite directives when the +intent is to overwrite a list.

+

This has the same behavior as a literal list, except that an +error will be triggered in the case that there is no underlying +list to overwrite; whereas a literal list will simply create a new +list.

+

The added error protection can be useful when intentionally +overwriting a list in an element’s public data, which is mostly +free form and not validated.

+

Example:

+
config:
+  install-commands:
+    # This element's `make install` is broken, replace it.
+    (=):
+    - cp src/program %{bindir}
+
+
+
+
+

(@) Include

+

Indicates that content should be loaded from files.

+

The include directive expects a string, or a list of strings when +including multiple files. Each of these strings represent a project +relative filename to include. Files can be included from subprojects +by prefixing the string with the locally defined junction +element and colon (‘:’).

+

The include directive can be used in any dictionary declared in the +project.conf, in any .bst file, or recursively included in another include file.

+

The including YAML fragment has priority over the files it includes, +and overrides any values introduced by the includes. When including +multiple files, files are included in the order they are declared in +the include list, and each subsequent include file takes priority over +the previous one.

+

Example:

+
environment:
+  (@): junction.bst:includes/environment.bst
+
+
+
+

Important

+

Files included across a junction cannot be used to inform the +declaration of a junction element, as +this can present a circular dependency.

+

Any variables, element +overrides, source +overrides or mirrors used in the declaration of a junction +must be declared in the project.conf or in +included files which are local to the project declaring the +junction itself.

+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/format_project.html b/format_project.html new file mode 100644 index 000000000..a7a060d4d --- /dev/null +++ b/format_project.html @@ -0,0 +1,1367 @@ + + + + + + + + + Project configuration — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Project configuration

+

The project configuration file should be named project.conf and +be located at the project root. It holds information such as Source +aliases relevant for the sources used in the given project as well as +overrides for the configuration of element types used in the project.

+

Values specified in the project configuration override any of the +default BuildStream project configuration, which is included +here for reference.

+
+

Essentials

+
+

Project name

+

The project name is a unique symbol for your project and will +be used to distinguish your project from others in user preferences, +namespacing of your project’s artifacts in shared artifact caches, +and in any case where BuildStream needs to distinguish between multiple +projects.

+

The first thing to setup in your project.conf should be the name +of your project.

+
name: my-project-name
+
+
+

The project name may contain alphanumeric characters, dashes and +underscores, and may not start with a leading digit.

+
+

Attention

+

The project name must be specified in the project.conf and +cannot be included from a separate file.

+
+
+
+

Minimum version

+

The BuildStream format is guaranteed to be backwards compatible +with any earlier minor point releases, which is to say that +BuildStream 1.4 can read projects written for BuildStream 1.0, +and that BuildStream 2.2 can read projects written for BuildStream 2.0.

+

Projects are required to specify the minimum version of BuildStream +which it requires, this allows project authors to convey a useful +error message to their users and peers, in the case that a user needs +to get a newer version of BuildStream in order to work with a given +project.

+

The project’s minimum required BuildStream version must be specified +in project.conf using the min-version field, e.g.:

+
# This project uses features which were added in 2.2
+min-version: 2.2
+
+
+

It is recommended that when using new features, always consult this +documentation and observe which BuildStream version a feature you are +using was added in. If a feature in the BuildStream YAML format is +not documented with a specific Since version, you can assume that +it has been there from the beginning.

+
+

Note

+

External Element and Source +plugins also implement their own YAML configuration fragments and as +such are revisioned separately from the core format.

+
+
+

Attention

+

The min-version must be specified in the project.conf and +cannot be included from a separate file.

+
+
+
+

Element path

+

To allow the user to structure their project nicely, BuildStream +allows the user to specify a project subdirectory where element +.bst files are stored.

+
element-path: elements
+
+
+

Note that elements are referred to by their relative paths, whenever +elements are referred to in a .bst file or on the command line.

+
+

Attention

+

The element-path can only be specified in the project.conf and +cannot be included from a separate file.

+
+
+
+

Ref storage

+

By default, BuildStream expects to read and write source references +directly in the source declaration, but this +can be inconvenient and prohibitive in some workflows.

+

Alternatively, BuildStream allows source references to be stored +centrally in a project.refs file in the toplevel +project directory.

+

This can be controlled with the ref-storage option, which is +allowed to be configured with the following values:

+
    +

  • inline

    +

    Source references are stored directly in the +source declaration

    +
    • +

  • project.refs

    +

    Source references are stored in the project.refs file, and +junction source references are stored in the junction.refs file.

    +
    • +
+

To enable storing of source references in project.refs, add the +following to your project.conf:

+
ref-storage: project.refs
+
+
+
+

Attention

+

Storing subproject source references in project.refs

+

When using the project.refs file, it is possible to override the +references in subprojects by editing the project.refs file directly +or by using bst source track –cross-junctions, +this can be practical to try out fresher versions of components which +are maintained in a subproject.

+

It should be noted however that overridden subproject source references listed +in your project.refs file will be ignored by projects which use your project +as a subproject.

+
+
+
+

Configurable Warnings

+

Warnings can be configured as fatal using the fatal-warnings configuration item. +When a warning is configured as fatal, where a warning would usually be thrown instead an error will be thrown +causing the build to fail.

+

Individual warnings can be configured as fatal by setting fatal-warnings to a list of warnings.

+
fatal-warnings:
+- overlaps
+- ref-not-in-track
+- <plugin>:<warning>
+
+
+

BuildStream provides a collection of Core Warnings which may be raised +by a variety of plugins. Other configurable warnings are plugin specific and should be noted within their individual documentation.

+
+
+

Source aliases

+

In order to abstract the download location of source code and +any assets which need to be downloaded, and also as a matter of +convenience, BuildStream allows one to create named aliases for +URLs which are to be used in the individual .bst files.

+
aliases:
+  foo: git://git.foo.org/
+  bar: http://bar.com/downloads/
+
+
+

If you want this project’s alias definitions to also be used for subprojects, +see Mapping source aliases of subprojects.

+
+
+

Sandbox options

+

Sandbox options for the whole project can be supplied in +project.conf in the same way as in an element. See element configuration +for more detail.

+
# Specify a user id and group id to use in the build sandbox.
+sandbox:
+  build-uid: 1003
+  build-gid: 1001
+
+
+
+
+

Artifact server

+

When maintaining a BuildStream project, it can be convenient to downstream users +of your project to provide access to a cache server you maintain.

+

The project can provide recommended artifact cache servers through project configuration +using the same semantics as one normally uses in the servers list of the +cache server user configuration:

+
#
+# A remote cache from which to download prebuilt artifacts
+#
+artifacts:
+- url: https://foo.com:11001
+  auth:
+    server-cert: server.crt
+
+
+
+

Attention

+

Unlike user configuration, the filenames provided in the auth +configuration block are relative to the project directory.

+

It is recommended to include public keys such as the server-cert along with your +project so that downstream users can have automatic read access to your project.

+

To provide write access to downstream users, it is recommended that the required +private keys such as the client-key be provided to users out of band, +and require that users configure write access separately in their own +user configuration.

+
+
+
+

Source cache server

+

In the same way as artifact cache servers, the project can provide recommended source cache +servers through project configuration using the same semantics as one normally uses in the +servers list of the cache server user configuration:

+
#
+# A remote cache from which to download prestaged sources
+#
+source-caches:
+- url: https://foo.com:11001
+  auth:
+    server-cert: server.crt
+
+
+
+

Attention

+

Unlike user configuration, the filenames provided in the auth +configuration block are relative to the project directory.

+

It is recommended to include public keys such as the server-cert along with your +project so that downstream users can have automatic read access to your project.

+

To provide write access to downstream users, it is recommended that the required +private keys such as the client-key be provided to users out of band, +and require that users configure write access separately in their own +user configuration.

+
+
+
+

Mirrors

+

A list of mirrors can be defined that couple a location to a mapping of aliases to a +list of URIs, e.g.

+
mirrors:
+- name: middle-earth
+  aliases:
+    foo:
+    - http://www.middle-earth.com/foo/1
+    - http://www.middle-earth.com/foo/2
+    bar:
+    - http://www.middle-earth.com/bar/1
+    - http://www.middle-earth.com/bar/2
+- name: oz
+  aliases:
+    foo:
+    - http://www.oz.com/foo
+    bar:
+    - http://www.oz.com/bar
+
+
+

The order that the mirrors (and the URIs therein) are consulted is in the order +they are defined when fetching, and in reverse-order when tracking.

+

The mirrors can be overridden on a per project basis using +user configuration. One can also specify which mirror should +be used first in the user configuration, or using +the –default-mirror command-line argument.

+

If you want this project’s mirrors to also be used for subprojects, +see Mapping source aliases of subprojects.

+
+
+
+

Loading plugins

+

If your project makes use of any custom Element or +Source plugins, then the project must inform BuildStream +of the plugins it means to make use of and the origin from which they can be loaded.

+

Note that plugins with the same name from different origins are not permitted.

+
+

Attention

+

The plugins can only be specified in the project.conf and cannot be +included from a separate file.

+
+
+

Local plugins

+

Local plugins are expected to be found in a subdirectory of the actual +BuildStream project. Element and +Source plugins should be stored in separate +directories to avoid namespace collisions, you can achieve this by +specifying a separate origin for sources and elements.

+
plugins:
+
+- origin: local
+  path: plugins/sources
+
+  # We want to use the `mysource` source plugin located in our
+  # project's `plugins/sources` subdirectory.
+  sources:
+  - mysource
+
+
+

There is no strict versioning policy for plugins loaded from the local +origin because the plugin is provided with the project data and as such, +it is considered to be completely deterministic.

+

Usually your project will be managed by a VCS like git, and any changes +to your local plugins may have an impact on your project, such as changes +to the artifact cache keys produced by elements which use these plugins. +Changes to plugins might provide new YAML configuration options, changes +in the semantics of existing configurations or even removal of existing +YAML configurations.

+
+
+

Pip plugins

+

Plugins loaded from the pip origin are expected to be installed +separately on the host operating system using python’s package management +system.

+
plugins:
+
+- origin: pip
+
+  # Specify the name of the python package containing
+  # the plugins we want to load. The name one would use
+  # on the `pip install` command line.
+  #
+  package-name: potato
+
+  # We again must specify specifically which plugins we
+  # want loaded from this origin.
+  #
+  elements:
+  - starch
+
+
+

Unlike local plugins, plugins loaded from the pip origin are +loaded from the active python environment, and as such you do not +usually have full control over the plugins your project uses unless +one uses strict version constraints.

+

The official plugin packages maintained by the BuildStream community are +guaranteed to be fully API stable. If one chooses to load these plugins +from the pip origin, then it is recommended to use minimal bound dependency +constraints when using +official plugin packages so as to be sure that you have access to all the +features you intend to use in your project.

+
+

Versioning constraints

+

When loading plugins from the pip plugin origin, it is possible to +specify constraints on the versions of packages you want to load +your plugins from.

+

The syntax for specifying versioning constraints is the same format supported by +the pip package manager.

+
+

Note

+

In order to be certain that versioning constraints work properly, plugin +packages should be careful to adhere to PEP 440, Version Identification and Dependency +Specification.

+
+

Here are a couple of examples:

+

Specifying minimal bound dependencies:

+
plugins:
+
+- origin: pip
+
+  # This project uses the API stable potato project and
+  # requires features from at least version 1.2
+  #
+  package-name: potato>=1.2
+
+
+

Specifying exact versions:

+
plugins:
+
+- origin: pip
+
+  # This project requires plugins from the potato
+  # project at exactly version 1.2.3
+  #
+  package-name: potato==1.2.3
+
+
+

Specifying version constraints:

+
plugins:
+
+- origin: pip
+
+  # This project requires plugins from the potato
+  # project from version 1.2.3 onward until 1.3.
+  #
+  package-name: potato>=1.2.3,<1.3
+
+
+
+

Important

+

Unstable plugin packages

+

When using unstable plugins loaded from the pip origin, the installed +plugins can sometimes be incompatible with your project.

+

Use virtual environments

+

Your operating system’s default python environment can only have one +version of a given package installed at a time, if you work on multiple +BuildStream projects on the same host, they may not agree on which versions +of plugins to use.

+

In order to guarantee that you can use a specific version of a plugin, +you may need to install BuildStream into a virtual environment in order to control which +python package versions are available when using your project.

+

Follow these instructions +to install BuildStream in a virtual environment.

+

Possible junction conflicts

+

If you have multiple projects which are connected through +junction elements, these projects can disagree +on which version of a plugin is needed from the pip origin.

+

Since only one version of a given plugin package can be installed +at a time in a given python environment, you must ensure that all +projects connected through junction elements +agree on which versions of API unstable plugin packages to use.

+
+
+
+
+

Junction plugins

+

Junction plugins are loaded from another project which your project has a +junction declaration for. Plugins are loaded directly +from the referenced project, the source and element plugins listed will simply +be loaded from the subproject regardless of how they were defined in that project.

+

Plugins loaded from a junction might even come from another junction and +be deeply nested.

+
plugins:
+
+- origin: junction
+
+  # Specify the local junction name declared in your
+  # project as the origin from where to load plugins from.
+  #
+  junction: subproject-junction.bst
+
+  # Here we want to get the `frobnicate` element
+  # from the subproject and use it in our project.
+  #
+  elements:
+  - frobnicate
+
+
+

Plugins loaded across junction boundaries will be loaded in the +context of your project, and any default values set in the project.conf +of the junctioned project will be ignored when resolving the +defaults provided with element plugins.

+

It is recommended to use include directives +in the case that the referenced plugins from junctioned projects depend +on variables defined in the project they come from, in this way you can include +variables needed by your plugins into your own project.conf.

+
+

Tip

+

Distributing plugins as projects

+

It is encouraged that people use BuildStream projects to distribute plugins +which are intended to be shared among projects, especially when these plugins +are not guaranteed to be completely API stable. This can still be done while +also distributing your plugins as pip packages at +the same time.

+

This can be achieved by simply creating a repository or tarball which +contains only the plugins you want to distribute, along with a project.conf +file declaring these plugins as local plugins.

+

Using plugins which are distributed as local plugins in a BuildStream project +ensures that you always have full control over which exact plugin your +project is using at all times, without needing to store the plugin as a +local plugin in your own project.

+
+
+
+

Suppressing deprecation warnings

+

Plugins can be deprecated over time, and using deprecated plugins will +trigger a warning when loading elements and sources which use +deprecated plugin kinds.

+

These deprecation warnings can be suppressed for the entire plugin +origin or on a per plugin kind basis.

+

To suppress all deprecation warnings from the origin, set the +allow-deprecated flag for the origin as follows:

+
plugins:
+
+- origin: local
+  path: plugins/sources
+
+  # Suppress deprecation warnings for any plugins loaded here
+  allow-deprecated: True
+
+  sources:
+  - mysource
+
+
+

In order to suppress deprecation warnings for a single element or +source kind within an origin, you will have to use a dictionary +to declare the specific plugin kind and set the allow-deprecated flag +on that dictionary as follows:

+
plugins:
+
+- origin: pip
+  package-name: potato
+
+  # Here we use a dictionary to declare the "starch"
+  # element kind, and specify that it is allowed to
+  # be deprecated.
+  #
+  elements:
+  - kind: starch
+    allow-deprecated: True
+
+
+
+
+
+

Options

+

Options are how BuildStream projects can define parameters which +can be configured by users invoking BuildStream to build your project.

+

Options are declared in the project.conf in the main options +dictionary.

+
options:
+  debug:
+    type: bool
+    description: Whether to enable debugging
+    default: False
+
+
+

Project options can be specified on the command line using +bst –option …

+
+

Note

+

The name of the option may contain alphanumeric characters +underscores, and may not start with a leading digit.

+
+
+

Common properties

+

All option types accept the following common attributes

+
    +

  • type

    +

    Indicates the type of option to declare

    +
    • +

  • description

    +

    A description of the meaning of the option

    +
    • +

  • variable

    +

    Optionally indicate a variable name to +export the option to. A string form of the selected option will +be used to set the exported value.

    +

    If used, this value will override any existing value for the +variable declared in project.conf, and will be overridden in +the regular composition order.

    +
    +

    Note

    +

    The name of the variable to export may contain alphanumeric +characters, dashes, underscores, and may not start with a leading +digit.

    +
    +
    • +
+
+
+

Boolean

+

The bool option type allows specifying boolean values which +can be cased in conditional expressions.

+

Declaring

+
options:
+  debug:
+    type: bool
+    description: Whether to enable debugging
+    default: False
+
+
+

Evaluating

+

Boolean options can be tested in expressions with equality tests:

+
variables:
+  enable-debug: False
+  (?):
+  - debug == True:
+      enable-debug: True
+
+
+

Or simply treated as truthy values:

+
variables:
+  enable-debug: False
+  (?):
+  - debug:
+      enable-debug: True
+
+
+

Exporting

+

When exporting boolean options as variables, a True option value +will be exported as 1 and a False option as 0

+
+
+

Enumeration

+

The enum option type allows specifying a string value +with a restricted set of possible values.

+

Declaring

+
options:
+  loglevel:
+    type: enum
+    description: The logging level
+    values:
+    - debug
+    - info
+    - warning
+    default: info
+
+
+

Evaluating

+

Enumeration options must be tested as strings in conditional +expressions:

+
variables:
+  enable-debug: False
+  (?):
+  - loglevel == "debug":
+      enable-debug: True
+
+
+

Exporting

+

When exporting enumeration options as variables, the value is +exported as a variable directly, as it is a simple string.

+
+
+

Flags

+

The flags option type allows specifying a list of string +values with a restricted set of possible values.

+

In contrast with the enum option type, the default value +need not be specified and will default to an empty set.

+

Declaring

+
options:
+  logmask:
+    type: flags
+    description: The logging mask
+    values:
+    - debug
+    - info
+    - warning
+    default:
+    - info
+
+
+

Evaluating

+

Options of type flags can be tested in conditional expressions using +a pythonic in syntax to test if an element is present in a set:

+
variables:
+  enable-debug: False
+  (?):
+  - ("debug" in logmask):
+      enable-debug: True
+
+
+

Exporting

+

When exporting flags options as variables, the value is +exported as a comma separated list of selected value strings.

+
+
+

Architecture

+

The arch option type is a special enumeration option which defaults via +uname -m results to the following list.

+
    +

  • aarch32

    • +

  • aarch64

    • +

  • aarch64-be

    • +

  • power-isa-be

    • +

  • power-isa-le

    • +

  • sparc-v9

    • +

  • x86-32

    • +

  • x86-64

    • +
+

The reason for this, opposed to using just uname -m, is that we want an +OS-independent list, as well as several results mapping to the same architecture +(e.g. i386, i486 etc. are all x86-32). It does not support assigning any default +in the project configuration.

+
options:
+  machine_arch:
+    type: arch
+    description: The machine architecture
+    values:
+    - aarch32
+    - aarch64
+    - x86-32
+    - x86-64
+
+
+

Architecture options can be tested with the same expressions +as other Enumeration options.

+
+
+

OS

+

The os option type is a special enumeration option, which defaults to the +results of uname -s. It does not support assigning any default in the project +configuration.

+
options:
+  machine_os:
+    type: os
+    description: The machine OS
+    values:
+    - Linux
+    - SunOS
+    - Darwin
+    - FreeBSD
+
+
+

Os options can be tested with the same expressions as other Enumeration options.

+
+
+

Element mask

+

The element-mask option type is a special Flags option +which automatically allows only element names as values.

+
options:
+  debug_elements:
+    type: element-mask
+    description: The elements to build in debug mode
+
+
+

This can be convenient for automatically declaring an option +which might apply to any element, and can be tested with the +same syntax as other Flag options.

+
variables:
+  enable-debug: False
+  (?):
+  - ("element.bst" in debug_elements):
+      enable-debug: True
+
+
+
+
+
+

Junctions

+

In this section of project.conf, we can define the relationship a project +has with junction elements in the same project, or +even in subprojects.

+

Sometimes when your project has multiple junction elements, +a situation can arise where you have multiple instances of the same project loaded +at the same time. In most cases, you will want to reconcile this conflict by ensuring +that your projects share the same junction. In order to reconcile conflicts by +ensuring nested junctions to the same project are shared, please refer to +the documentation on nested junctions.

+

In some exceptional cases, it is entirely intentional and appropriate to use +the same project more than once in the same build pipeline. The attributes +in the junctions group here in project.conf provide some tools you can +use to explicitly allow the coexistence of the same project multiple times.

+
+

Duplicate junctions

+

In the case that you are faced with an error due to subprojects sharing +a common sub-subproject, you can use the duplicates configuration +in order to allow the said project to be loaded twice.

+

Example:

+
junctions:
+
+  duplicates:
+
+    # Here we use the packaging tooling completely separately from
+    # the payload that we are packaging, they are never staged to
+    # the same location in a given sandbox, and as such we would
+    # prefer to allow the 'runtime' project to be loaded separately.
+    #
+    # This statement will ensure that loading the 'runtime' project
+    # from these two locations will not produce any errors.
+    #
+    runtime:
+    - payload.bst:runtime.bst
+    - packaging.bst:runtime.bst
+
+
+

When considering duplicated projects in the same pipeline, all instances +of the said project need to be marked as duplicates in order to avoid +a conflicting junction error at load time.

+
+

Tip

+

The declaration of duplicates is inherited by any dependant projects +which may later decide to depend on your project.

+

If you depend on a project which itself has duplicates, and you need +to duplicate it again, then you only need to declare the new duplicate, +you do not need to redeclare duplicates redundantly.

+
+
+
+

Internal junctions

+

Another way to avoid conflicting junction errors when you know that your +subproject should not conflict with other instances of the same subproject, +is to declare the said subproject as internal.

+

Example:

+
junctions:
+
+  # Declare this subproject as "internal" because we know
+  # that we only use it for build dependencies, and as such
+  # we know that it cannot collide with elements in dependant
+  # projects.
+  #
+  internal:
+  - special-compiler.bst
+
+
+

When compared to duplicates above, internal projects have the advantage +of never producing any conflicting junction errors in dependant projects +(reverse dependency projects).

+

This approach is preferrable in cases where you know for sure that dependant +projects will not be depending directly on elements from your internal +subproject.

+
+

Attention

+

Declaring a junction as internal is a promise that dependant projects +will not accrue runtime dependencies on elements in your internal subproject.

+
+
+
+

Mapping source aliases of subprojects

+

junction elements allow source aliases of subprojects +to be mapped to aliases of the parent project. This makes it possible to control +the translation of aliases to URLs including mirror configuration across multiple +project levels.

+

To ensure that there are mappings for all aliases of all subprojects, you can set the +disallow-subproject-uris flag in the junctions group here in project.conf.

+

top-level

+
junctions:
+  disallow-subproject-uris: True
+
+
+

This will raise an error if an alias without a mapping is encountered. This flag +is applied recursively across all junctions.

+

It also configures unaliased-url as a fatal warning in all subprojects to +ensure that the current project is in full control over all source URLs. +As the fatal warning configuration contributes to the cache key, this flag will +affect the cache key of subprojects that haven’t already configured +unaliased-url as a fatal warning.

+
+
+
+

Element default configuration

+

The project.conf plays a role in defining elements by +providing default values and also by overriding values declared +by plugins on a plugin wide basis.

+

See the composition documentation for +more detail on how elements are composed.

+
+

Variables

+

The defaults for Variables used in your +project is defined here.

+
variables:
+  prefix: "/usr"
+
+
+
+
+

Environment

+

The defaults environment for the build sandbox is defined here.

+
environment:
+  PATH: /usr/bin:/bin:/usr/sbin:/sbin
+
+
+

Additionally, the special environment-nocache list which specifies +which environment variables do not affect build output, and are thus +not considered in the calculation of artifact keys can be defined here.

+
environment-nocache:
+- MAXJOBS
+
+
+

Note that the environment-nocache list only exists so that we can +control parameters such as make -j ${MAXJOBS}, allowing us to control +the number of jobs for a given build without affecting the resulting +cache key.

+
+
+

Split rules

+

The project wide split rules defaults can +be specified here.

+
split-rules:
+  devel:
+  - |
+    %{includedir}
+  - |
+    %{includedir}/**
+  - |
+    %{libdir}/lib*.a
+  - |
+    %{libdir}/lib*.la
+
+
+
+
+
+

Overriding plugin defaults

+

Base attributes declared by element and source plugins can be overridden +on a project wide basis. This section explains how to make project wide +statements which augment the configuration of an element or source plugin.

+
+

Element overrides

+

The elements dictionary can be used to override variables, environments +or plugin specific configuration data as shown below.

+
elements:
+
+  # Override default values for all autotools elements
+  autotools:
+
+    variables:
+      bindir: "%{prefix}/bin"
+
+    config:
+      configure-commands: ...
+
+    environment:
+      PKG_CONFIG_PATH=%{libdir}/pkgconfig
+
+
+
+
+

Source overrides

+

The sources dictionary can be used to override source plugin +specific configuration data as shown below.

+
sources:
+
+  # Override default values for all git sources
+  git:
+
+    config:
+      checkout-submodules: False
+
+
+
+
+
+

Customizing the shell

+

Since BuildStream cannot know intimate details about your host or about +the nature of the runtime and software that you are building, the shell +environment for debugging and testing applications may need some help.

+

The shell section allows some customization of the shell environment.

+
+

Interactive shell command

+

By default, BuildStream will use sh -i when running an interactive +shell, unless a specific command is given to the bst shell command.

+

BuildStream will automatically set a convenient prompt via the PS1 +environment variable for interactive shells; which might be overwritten +depending on the shell you use in your runtime.

+

If you are using bash, we recommend the following configuration to +ensure that the customized prompt is not overwritten:

+
shell:
+
+  # Specify the command to run by default for interactive shells
+  command: [ 'bash', '--noprofile', '--norc', '-i' ]
+
+
+
+
+

Environment assignments

+

In order to cooperate with your host environment, a debugging shell +sometimes needs to be configured with some extra knowledge inheriting +from your host environment.

+

This can be achieved by setting up the shell environment configuration, +which is expressed as a dictionary very similar to the +default environment, except that it +supports host side environment variable expansion in values.

+

For example, to share your host DISPLAY and DBUS_SESSION_BUS_ADDRESS +environments with debugging shells for your project, specify the following:

+
shell:
+
+  # Share some environment variables from the host environment
+  environment:
+    DISPLAY: '$DISPLAY'
+    DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS'
+
+
+

Or, a more complex example is how one might share the host pulseaudio +server with a bst shell environment:

+
shell:
+
+  # Set some environment variables explicitly
+  environment:
+    PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native'
+
+
+
+
+

Host files

+

It can be useful to share some files on the host with a shell so that +it can integrate better with the host environment.

+

The host-files configuration allows one to specify files and +directories on the host to be bind mounted into the sandbox.

+
+

Warning

+

One should never mount directories where one expects to +find data and files which belong to the user, such as /home +on POSIX platforms.

+

This is because the unsuspecting user may corrupt their own +files accidentally as a result. Instead users can use the +--mount option of bst shell to mount data into the shell.

+
+

The host-files configuration is an ordered list of mount specifications.

+

Members of the list can be fully specified as a dictionary, or a simple +string can be used if only the defaults are required.

+

The fully specified dictionary has the following members:

+
    +

  • path

    +

    The path inside the sandbox. This is the only mandatory +member of the mount specification.

    +
    • +

  • host_path

    +

    The host path to mount at path in the sandbox. This +will default to path if left unspecified.

    +
    • +

  • optional

    +

    Whether the mount should be considered optional. This +is False by default.

    +
    • +
+

Here is an example of a fully specified mount specification:

+
shell:
+
+  # Mount an arbitrary resolv.conf from the host to
+  # /etc/resolv.conf in the sandbox, and avoid any
+  # warnings if the host resolv.conf doesnt exist.
+  host-files:
+  - host_path: '/usr/local/work/etc/resolv.conf'
+    path: '/etc/resolv.conf'
+    optional: True
+
+
+

Here is an example of using shorthand mount specifications:

+
shell:
+
+  # Specify a list of files to mount in the sandbox
+  # directory from the host.
+  #
+  # If these do not exist on the host, a warning will
+  # be issued but the shell will still be launched.
+  host-files:
+  - '/etc/passwd'
+  - '/etc/group'
+  - '/etc/resolv.conf'
+
+
+

Host side environment variable expansion is also supported:

+
shell:
+
+  # Mount a host side pulseaudio server socket into
+  # the shell environment at the same location.
+  host-files:
+  - '${XDG_RUNTIME_DIR}/pulse/native'
+
+
+
+
+
+

Default targets

+

When running BuildStream commands from a project directory or subdirectory +without specifying any target elements on the command line, the default targets +of the project will be used. The default targets can be configured in the +defaults section as follows:

+
defaults:
+
+  # List of default target elements
+  targets:
+  - app.bst
+
+
+

If no default targets are configured in project.conf, BuildStream commands +will default to all .bst files in the configured element path.

+

Commands that cannot support junctions as target elements (bst build, +bst artifact push, and bst artifact pull) ignore junctions in the list +of default targets.

+

When running BuildStream commands from a workspace directory (that is not a +BuildStream project directory), project default targets are not used and the +workspace element will be used as the default target instead.

+

bst artifact checkout, bst source checkout, and bst shell are +currently limited to a single target element and due to this, they currently +do not use project default targets. However, they still use the workspace +element as default target when run from a workspace directory.

+
+
+

Builtin defaults

+

BuildStream defines some default values for convenience, the default +values overridden by your project’s project.conf are presented here:

+
+
#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# Default BuildStream project configuration.
+
+
+# General configuration defaults
+#
+
+# Elements are found at the project root
+element-path: .
+
+# Store source references in element files
+ref-storage: inline
+
+# Variable Configuration
+#
+variables:
+  # Path configuration, to be used in build instructions.
+  prefix: "/usr"
+  exec_prefix: "%{prefix}"
+  bindir: "%{exec_prefix}/bin"
+  sbindir: "%{exec_prefix}/sbin"
+  libexecdir: "%{exec_prefix}/libexec"
+  datadir: "%{prefix}/share"
+  sysconfdir: "/etc"
+  sharedstatedir: "%{prefix}/com"
+  localstatedir: "/var"
+  lib: "lib"
+  libdir: "%{prefix}/%{lib}"
+  debugdir: "%{libdir}/debug"
+  includedir: "%{prefix}/include"
+  docdir: "%{datadir}/doc"
+  infodir: "%{datadir}/info"
+  mandir: "%{datadir}/man"
+
+  # Indicates the default build directory where input is
+  # normally staged
+  build-root: /buildstream/%{project-name}/%{element-name}
+
+  # Indicates where the build system should look for configuration files  
+  conf-root: .
+  
+  # Indicates the build installation directory in the sandbox
+  install-root: /buildstream-install
+
+  # You need to override this with the commands specific for your system
+  strip-binaries: ""
+
+# Base sandbox environment, can be overridden by plugins
+environment:
+  PATH: /usr/bin:/bin:/usr/sbin:/sbin
+  SHELL: /bin/sh
+  TERM: dumb
+  USER: tomjon
+  USERNAME: tomjon
+  LOGNAME: tomjon
+  LC_ALL: C
+  HOME: /tmp
+  TZ: UTC
+
+  # For reproducible builds we use 2011-11-11 11:11:11 UTC as a constant
+  SOURCE_DATE_EPOCH: 1321009871
+
+# List of environment variables which should not be taken into
+# account when calculating a cache key for a given element.
+#
+environment-nocache: []
+
+# Configuration for the sandbox other than environment variables
+# should go in 'sandbox'.
+sandbox: {}
+
+# Defaults for the 'split-rules' public data found on elements
+# in the 'bst' domain.
+#
+split-rules:
+
+  # The runtime domain includes whatever is needed for the
+  # built element to run, this includes stripped executables
+  # and shared libraries by default.
+  runtime:
+  - |
+    %{bindir}
+  - |
+    %{bindir}/*
+  - |
+    %{sbindir}
+  - |
+    %{sbindir}/*
+  - |
+    %{libexecdir}
+  - |
+    %{libexecdir}/*
+  - |
+    %{libdir}/lib*.so*
+
+  # The devel domain includes additional things which
+  # you may need for development.
+  #
+  # By default this includes header files, static libraries
+  # and other metadata such as pkgconfig files, m4 macros and
+  # libtool archives.
+  devel:
+  - |
+    %{includedir}
+  - |
+    %{includedir}/**
+  - |
+    %{libdir}/lib*.a
+  - |
+    %{libdir}/lib*.la
+  - |
+    %{libdir}/pkgconfig/*.pc
+  - |
+    %{datadir}/pkgconfig/*.pc
+  - |
+    %{datadir}/aclocal/*.m4
+
+  # The debug domain includes debugging information stripped
+  # away from libraries and executables
+  debug:
+  - |
+    %{debugdir}
+  - |
+    %{debugdir}/**
+
+  # The doc domain includes documentation
+  doc:
+  - |
+    %{docdir}
+  - |
+    %{docdir}/**
+  - |
+    %{infodir}
+  - |
+    %{infodir}/**
+  - |
+    %{mandir}
+  - |
+    %{mandir}/**
+
+  # The locale domain includes translations etc
+  locale:
+  - |
+    %{datadir}/locale
+  - |
+    %{datadir}/locale/**
+  - |
+    %{datadir}/i18n
+  - |
+    %{datadir}/i18n/**
+  - |
+    %{datadir}/zoneinfo
+  - |
+    %{datadir}/zoneinfo/**
+
+
+# Default behavior for `bst shell`
+#
+shell:
+
+  # Command to run when `bst shell` does not provide a command
+  #
+  command: [ 'sh', '-i' ]
+
+# Defaults for bst commands
+#
+defaults:
+
+  # Set default target elements to use when none are passed on the command line.
+  # If none are configured in the project, default to all project elements.
+  targets: []
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/format_project_refs.html b/format_project_refs.html new file mode 100644 index 000000000..39dafbbc7 --- /dev/null +++ b/format_project_refs.html @@ -0,0 +1,194 @@ + + + + + + + + + The project.refs file — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

The project.refs file

+

If one has elected to store source references in a single project.refs +file, then it will be stored at the toplevel of your project directory +adjacent to project.conf. This can be configured in your project +using the ref-storage configuration

+

Sources for junction elements are stored +separately in an adjacent junction.refs file of the same format.

+
+

Basic behavior

+

When a project.refs file is in use, any source references found +in the inline source declarations are considered +invalid and will be ignored, and a warning will be emitted for them.

+

When bst source track is run for your project, the project.refs file +will be updated instead of the inline source declarations. In the absence +of a project.refs file, bst source track will create one automatically +with the tracking results.

+

An interesting property of project.refs is that it allows for +cross junction tracking. This is to say that it is possible to override +the ref of a given source in a project that your project depends on via +a junction, without actually modifying the +junctioned project.

+
+
+

Format

+

The project.refs uses the same YAML format used throughout BuildStream, +and supports the same directives which apply to +project.conf and element declaration files (i.e. element.bst files).

+

The project.refs file format itself is very simple, it contains a single projects +key at the toplevel, which is a dictionary of project names. +Each project name is a dictionary of element names, and each element name holds +a list of dictionaries corresponding to the element’s sources.

+

Example

+
# Main toplevel "projects" key
+projects:
+
+  # The local project's name is "core"
+  core:
+
+    # A dictionary of element names
+    base/automake.bst:
+
+    # A list of sources corresponding to the element
+    # in the same order in which they were declared.
+    #
+    # The values of this list are dictionaries of the
+    # symbolic "ref" portion understood by the given
+    # source plugin implementation.
+    #
+    - ref: af6ba39142220687c500f79b4aa2f181d9b24e4...
+
+  # The "core" project depends on the "bootstrap" project,
+  # here we are allowed to override the refs for the projects
+  # we depend on through junctions.
+  bootstrap:
+
+    zlib.bst:
+    - ref: 4ff941449631ace0d4d203e3483be9dbc9da4540...
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/format_public.html b/format_public.html new file mode 100644 index 000000000..8cded77a2 --- /dev/null +++ b/format_public.html @@ -0,0 +1,211 @@ + + + + + + + + + Builtin public data — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Builtin public data

+

Elements can provide public data which can be read by other elements +later in the pipeline, the format for exposing public data on a given +element is described here.

+

Any element may use public data for whatever purpose it wants, but +BuildStream has some built-in expectations of public data, which resides +completely in the bst domain.

+

In this section we will describe the public data in the bst domain.

+
+

Integration commands

+
# Specify some integration commands
+public:
+  bst:
+    integration-commands:
+    - /usr/bin/update-fancy-feature-cache
+
+
+

The built-in integration-commands list indicates that depending elements +should run this set of commands before expecting the staged runtime environment +to be functional.

+

Typical cases for this include running ldconfig at the base of a pipeline, +or running commands to update various system caches.

+

Integration commands of a given element are automatically run by the +Element.integrate() method +and are used by various plugins.

+

Notably the BuildElement derived classes +will always integrate the build dependencies after staging and before running +any build commands.

+
+
+

Split rules

+
# Specify some split rules
+public:
+  bst:
+    split-rules:
+      runtime:
+      - |
+        %{bindir}/*
+      - |
+        %{sbindir}/*
+      - |
+        %{libexecdir}/*
+      - |
+        %{libdir}/lib*.so*
+
+
+

Split rules indicate how the output of an element can be categorized +into domains.

+

The split-rules domains are used by the +Element.stage_artifact() +method when deciding what domains of an artifact should be staged.

+

The strings listed in each domain are first substituted with the +variables in context of the given element, and +then applied as a glob style match, as understood by +utils.glob()

+

This is used for creating compositions with the compose +element and can be used by other deployment related elements for the purpose of +splitting element artifacts into separate packages.

+
+
+

Overlap whitelist

+

The overlap whitelist indicates which files this element is allowed to overlap +over other elements when staged together with other elements.

+

Each item in the overlap whitelist has substitutions applied from +variables, and is then applied as a glob-style match +(i.e. utils.glob()).

+
public:
+  bst:
+    overlap-whitelist:
+    - |
+      %{sysconfdir}/*
+    - |
+      /etc/fontcache
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/genindex.html b/genindex.html new file mode 100644 index 000000000..57800bb3c --- /dev/null +++ b/genindex.html @@ -0,0 +1,2231 @@ + + + + + + + + Index — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    +
    • +
  • Index
    • +
  • +
    • +
+
+
+
+
+ + +

Index

+ +
+ Symbols + | A + | B + | C + | D + | E + | F + | G + | I + | J + | K + | L + | M + | N + | O + | P + | R + | S + | T + | U + | V + | W + +
+

Symbols

+ + + +
+ +

A

+ + + +
+ +

B

+ + + +
+ +

C

+ + + +
+ +

D

+ + + +
+ +

E

+ + + +
+ +

F

+ + + +
+ +

G

+ + + +
+ +

I

+ + + +
+ +

J

+ + +
+ +

K

+ + +
+ +

L

+ + + +
+ +

M

+ + + +
+ +

N

+ + + +
+ +

O

+ + + +
+ +

P

+ + + +
+ +

R

+ + + +
+ +

S

+ + + +
+ +

T

+ + + +
+ +

U

+ + + +
+ +

V

+ + + +
+ +

W

+ + + +
+ + + +
+
+
+ +
+ +
+

© Copyright 2017-2022, The Apache Software Foundation.

+
+ + Built with Sphinx using a + theme + provided by Read the Docs. + + +
+
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/coding_guidelines.html b/hacking/coding_guidelines.html new file mode 100644 index 000000000..80ed64e8d --- /dev/null +++ b/hacking/coding_guidelines.html @@ -0,0 +1,935 @@ + + + + + + + + + Coding guidelines — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Coding guidelines

+

This section discusses coding style and other guidelines for hacking +on BuildStream. This is important to read through for writing any non-trivial +patches and especially outlines what people should watch out for when +reviewing patches.

+

Much of the rationale behind what is layed out in this section considers +good traceability of lines of code with git blame, overall sensible +modular structure, consistency in how we write code, and long term maintenance +in mind.

+
+

Approximate PEP-8 Style

+

Python coding style for BuildStream is approximately pep8.

+

The coding style is automatically enforced by black.

+

Formatting will be checked automatically when running the testsuite on CI. For +details on how to format your code locally, see formatting code.

+
+
+

Documenting symbols

+

In BuildStream, we maintain what we call a “Public API Surface” that +is guaranteed to be stable and unchanging across stable releases. The +symbols which fall into this special class are documented using Python’s +standard docstrings, while all other internals of BuildStream are documented +with comments above the related symbol.

+

When documenting the public API surface which is rendered in the reference +manual, we always mention the major version in which the API was introduced, +as shown in the examples below. If a public API exists without the Since +annotation, this is taken to mean that it was available since the first stable +major point release (e.g: 2.0).

+

We also always ensure that the public API is entirely typed using type +annotations inline.

+

The private API can be typed inline or in the documentation at the author’s +discretion.

+
+

Note

+

Types are checked using mypy. You can run it like tox -e mypy

+
+

Here are some examples to get the hang of the format of API documenting +comments and docstrings.

+

Public API Surface method:

+
def frobnicate(self, source: Source, *, frobilicious: bool = False) -> Element:
+    """Frobnicates this element with the specified source
+
+    Args:
+       source: The Source to frobnicate with
+       frobilicious: Optionally specify that frobnication should be
+                            performed frobiliciously
+
+    Returns:
+       The frobnicated version of this Element.
+
+    *Since: 2.2*
+    """
+    ...
+
+
+

Internal method:

+
# frobnicate():
+#
+# Frobnicates this element with the specified source
+#
+# Args:
+#    source: The Source to frobnicate with
+#    frobilicious: Optionally specify that frobnication should be
+#                         performed frobiliciously
+#
+# Returns:
+#    The frobnicated version of this Element.
+#
+def frobnicate(self, source: Source, *, frobilicious: bool = False) -> Element:
+    ...
+
+
+

Public API Surface instance variable:

+
def __init__(self, context, element):
+
+  self.name = self._compute_name(context, element)
+  """The name of this foo
+
+  *Since: 2.2*
+  """
+
+
+
+

Note

+

Python does not support docstrings on instance variables, but sphinx does +pick them up and includes them in the generated documentation.

+
+

Internal instance variable:

+
def __init__(self, context, element):
+
+  self.name = self._compute_name(context, element)  # The name of this foo
+
+
+

Internal instance variable (long):

+
def __init__(self, context, element):
+
+  # This instance variable required a longer explanation, so
+  # it is on a line above the instance variable declaration.
+  self.name = self._compute_name(context, element)
+
+
+

Public API Surface class:

+
class Foo(Bar):
+    """The main Foo object in the data model
+
+    Explanation about Foo. Note that we always document
+    the constructor arguments here, and not beside the __init__
+    method.
+
+    Args:
+       context: The invocation Context
+       count: The number to count
+
+    *Since: 2.2*
+    """
+    def __init__(self, context: Context, count: int) -> None:
+    ...
+
+
+

Internal class:

+
# Foo()
+#
+# The main Foo object in the data model
+#
+# Args:
+#    context (Context): The invocation Context
+#    count (int): The number to count
+#
+class Foo(Bar):
+    ...
+
+
+
+
+

Class structure and ordering

+

When creating or modifying an object class in BuildStream, it is +important to keep in mind the order in which symbols should appear +and keep this consistent.

+

Here is an example to illustrate the expected ordering of symbols +on a Python class in BuildStream:

+
class Foo(Bar):
+
+    # Public class-wide variables come first, if any.
+
+    # Private class-wide variables, if any
+
+    # Now we have the dunder/magic methods, always starting
+    # with the __init__() method.
+
+    def __init__(self, name):
+
+        super().__init__()
+
+        # NOTE: In the instance initializer we declare any instance variables,
+        #       always declare the public instance variables (if any) before
+        #       the private ones.
+        #
+        #       It is preferred to avoid any public instance variables, and
+        #       always expose an accessor method for it instead.
+
+        #
+        # Public instance variables
+        #
+        self.name = name  # The name of this foo
+
+        #
+        # Private instance variables
+        #
+        self._count = 0   # The count of this foo
+
+    ################################################
+    #               Abstract Methods               #
+    ################################################
+
+    # NOTE: Abstract methods in BuildStream are allowed to have
+    #       default methods.
+    #
+    #       Subclasses must NEVER override any method which was
+    #       not advertized as an abstract method by the parent class.
+
+    # frob()
+    #
+    # Implementors should implement this to frob this foo
+    # count times if possible.
+    #
+    # Args:
+    #    count (int): The number of times to frob this foo
+    #
+    # Returns:
+    #    (int): The number of times this foo was frobbed.
+    #
+    # Raises:
+    #    (FooError): Implementors are expected to raise this error
+    #
+    def frob(self, count):
+
+        #
+        # An abstract method in BuildStream is allowed to have
+        # a default implementation.
+        #
+        self._count = self._do_frobbing(count)
+
+        return self._count
+
+    ################################################
+    #     Implementation of abstract methods       #
+    ################################################
+
+    # NOTE: Implementations of abstract methods defined by
+    #       the parent class should NEVER document the API
+    #       here redundantly.
+
+    def frobbish(self):
+       #
+       # Implementation of the "frobbish" abstract method
+       # defined by the parent Bar class.
+       #
+       return True
+
+    ################################################
+    #                 Public Methods               #
+    ################################################
+
+    # NOTE: Public methods here are the ones which are expected
+    #       to be called from outside of this class.
+    #
+    #       These, along with any abstract methods, usually
+    #       constitute the API surface of this class.
+
+    # frobnicate()
+    #
+    # Perform the frobnication process on this Foo
+    #
+    # Raises:
+    #    (FrobError): In the case that a frobnication error was
+    #                 encountered
+    #
+    def frobnicate(self):
+        frobnicator.frobnicate(self)
+
+    # set_count()
+    #
+    # Sets the count of this foo
+    #
+    # Args:
+    #    count (int): The new count to set
+    #
+    def set_count(self, count):
+
+        self._count = count
+
+    # get_count()
+    #
+    # Accessor for the count value of this foo.
+    #
+    # Returns:
+    #    (int): The count of this foo
+    #
+    def get_count(self, count):
+
+        return self._count
+
+    ################################################
+    #                 Private Methods              #
+    ################################################
+
+    # NOTE: Private methods are the ones which are internal
+    #       implementation details of this class.
+    #
+    #       Even though these are private implementation
+    #       details, they still MUST have API documenting
+    #       comments on them.
+
+    # _do_frobbing()
+    #
+    # Does the actual frobbing
+    #
+    # Args:
+    #    count (int): The number of times to frob this foo
+    #
+    # Returns:
+    #    (int): The number of times this foo was frobbed.
+    #
+    def self._do_frobbing(self, count):
+        return count
+
+
+
+
+

Public and private symbols

+

BuildStream mostly follows the PEP-8 for defining public and private symbols +for any given class, with some deviations. Please read the section on inheritance for +reference on how the PEP-8 defines public and non-public.

+
    +

  • A public symbol is any symbol which you expect to be used by clients +of your class or module within BuildStream.

    +

    Public symbols are written without any leading underscores.

    +
    • +

  • A private symbol is any symbol which is entirely internal to your class +or module within BuildStream. These symbols cannot ever be accessed by +external clients or modules.

    +

    A private symbol must be denoted by a leading underscore.

    +
    • +

  • When a class can have subclasses, then private symbols should be denoted +by two leading underscores. For example, the Sandbox or Platform +classes which have various implementations, or the Element and Source +classes which plugins derive from.

    +

    The double leading underscore naming convention invokes Python’s name +mangling algorithm which helps prevent namespace collisions in the case +that subclasses might have a private symbol with the same name.

    +
    • +
+

In BuildStream, we have what we call a “Public API Surface”, as previously +mentioned in Documenting symbols. In the next section we will discuss the “Public API Surface” and +outline the exceptions to the rules discussed here.

+
+
+

Public API surface

+

BuildStream exposes what we call a “Public API Surface” which is stable +and unchanging. This is for the sake of stability of the interfaces which +plugins use, so it can also be referred to as the “Plugin facing API”.

+

Any symbols which are a part of the “Public API Surface” are never allowed +to change once they have landed in a stable release version of BuildStream. As +such, we aim to keep the “Public API Surface” as small as possible at all +times, and never expose any internal details to plugins inadvertently.

+

One problem which arises from this is that we end up having symbols +which are public according to the rules discussed in the previous section, but must be hidden away from the +“Public API Surface”. For example, BuildStream internal classes need +to invoke methods on the Element and Source classes, whereas these +methods need to be hidden from the “Public API Surface”.

+

This is where BuildStream deviates from the PEP-8 standard for public +and private symbol naming.

+

In order to disambiguate between:

+
    +

  • Symbols which are publicly accessible details of the Element class, can +be accessed by BuildStream internals, but must remain hidden from the +“Public API Surface”.

    • +

  • Symbols which are private to the Element class, and cannot be accessed +from outside of the Element class at all.

    • +
+

We denote the former category of symbols with only a single underscore, and the latter +category of symbols with a double underscore. We often refer to this distinction +as “API Private” (the former category) and “Local Private” (the latter category).

+

Classes which are a part of the “Public API Surface” and require this disambiguation +were not discussed in the class ordering section, for +these classes, the “API Private” symbols always come before the “Local Private” +symbols in the class declaration.

+

Modules which are not a part of the “Public API Surface” have their Python files +prefixed with a single underscore, and are not imported in BuildStream’s the master +__init__.py which is used by plugins.

+
+

Note

+

The utils.py module is public and exposes a handful of utility functions, +however many of the functions it provides are “API Private”.

+

In this case, the “API Private” functions are prefixed with a single underscore.

+
+

Any objects which are a part of the “Public API Surface” should be exposed via the +toplevel __init__.py of the buildstream package.

+
+
+

File naming convention

+

With the exception of a few helper objects and data structures, we structure +the code in BuildStream such that every filename is named after the object it +implements. E.g. The Project object is implemented in _project.py, the +Context object in _context.py, the base Element class in element.py, +etc.

+

As mentioned in the previous section, objects which are not a part of the +public, plugin facing API surface have their +filenames prefixed with a leading underscore (like _context.py and _project.py +in the examples above).

+

When an object name has multiple words in it, e.g. ArtifactCache, then the +resulting file is named all in lower case without any underscore to separate +words. In the case of ArtifactCache, the filename implementing this object +is found at _artifactcache/artifactcache.py.

+
+
+

Imports

+

Module imports inside BuildStream are done with relative . notation:

+

Good:

+
from ._context import Context
+
+
+

Bad:

+
from buildstream._context import Context
+
+
+

The exception to the above rule is when authoring plugins, +plugins do not reside in the same namespace so they must +address buildstream in the imports.

+

An element plugin will derive from Element by importing:

+
from buildstream import Element
+
+
+

When importing utilities specifically, don’t import function names +from there, instead import the module itself:

+
from . import utils
+
+
+

This makes things clear when reading code that said functions +are not defined in the same file but come from utils.py for example.

+
+
+

Instance variables

+

It is preferred that all instance state variables be declared as private symbols, however in some cases, especially when the state +is immutable for the object’s life time (like an Element name for example), it +is acceptable to save some typing by using a publicly accessible instance variable.

+

It is never acceptable to modify the value of an instance variable from outside +of the declaring class, even if the variable is public. In other words, the class +which exposes an instance variable is the only one in control of the value of this +variable.

+
    +

  • If an instance variable is public and must be modified; then it must be +modified using a mutator.

    • +

  • Ideally for better encapsulation, all object state is declared as +private instance variables and can +only be accessed by external classes via public accessors and mutators.

    • +
+
+

Note

+

In some cases, we may use small data structures declared as objects for the sake +of better readability, where the object class itself has no real supporting code.

+

In these exceptions, it can be acceptable to modify the instance variables +of these objects directly, unless they are otherwise documented to be immutable.

+
+
+
+

Accessors and mutators

+

An accessor and mutator, are methods defined on the object class to access (get) +or mutate (set) a value owned by the declaring class, respectively.

+

An accessor might derive the returned value from one or more of its components, +and a mutator might have side effects, or delegate the mutation to a component.

+

Accessors and mutators are always public +(even if they might have a single leading underscore and are considered +API Private), as their purpose is to +enforce encapsulation with regards to any accesses to the state which is owned +by the declaring class.

+

Accessors and mutators are functions prefixed with get_ and set_ +respectively, e.g.:

+
class Foo():
+
+    def __init__(self):
+
+        # Declare some internal state
+        self._count = 0
+
+    # get_count()
+    #
+    # Gets the count of this Foo.
+    #
+    # Returns:
+    #    (int): The current count of this Foo
+    #
+    def get_foo(self):
+        return self._count
+
+    # set_count()
+    #
+    # Sets the count of this Foo.
+    #
+    # Args:
+    #    count (int): The new count for this Foo
+    #
+    def set_foo(self, count):
+        self._count = count
+
+
+
+

Attention

+

We are aware that Python offers a facility for accessors and +mutators using the @property decorator instead. Do not use +the @property decorator.

+

The decision to use explicitly defined functions instead of the +@property decorator is rather arbitrary, there is not much +technical merit to preferring one technique over the other. +However as discussed below, +it is of the utmost importance that we do not mix both techniques +in the same codebase.

+
+
+
+

Abstract methods

+

In BuildStream, an “Abstract Method” is a bit of a misnomer and does +not match up to how Python defines abstract methods, we need to seek out +a new nomenclature to refer to these methods.

+

In Python, an “Abstract Method” is a method which must be +implemented by a subclass, whereas all methods in Python can be +overridden.

+

In BuildStream, we use the term “Abstract Method”, to refer to +a method which can be overridden by a subclass, whereas it +is illegal to override any other method.

+
    +

  • Abstract methods are allowed to have default implementations.

    • +

  • Subclasses are not allowed to redefine the calling signature +of an abstract method, or redefine the API contract in any way.

    • +

  • Subclasses are not allowed to override any other methods.

    • +
+

The key here is that in BuildStream, we consider it unacceptable +that a subclass overrides a method of its parent class unless +the said parent class has explicitly given permission to subclasses +to do so, and outlined the API contract for this purpose. No surprises +are allowed.

+
+
+

Error handling

+

In BuildStream, all non recoverable errors are expressed via +subclasses of the BstError exception.

+

This exception is handled deep in the core in a few places, and +it is rarely necessary to handle a BstError.

+
+

Raising exceptions

+

When writing code in the BuildStream core, ensure that all system +calls and third party library calls are wrapped in a try: block, +and raise a descriptive BstError of the appropriate class explaining +what exactly failed.

+

Ensure that the original system call error is formatted into your new +exception, and that you use the Python from semantic to retain the +original call trace, example:

+
try:
+    os.utime(self._refpath(ref))
+except FileNotFoundError as e:
+    raise ArtifactError("Attempt to access unavailable artifact: {}".format(e)) from e
+
+
+
+
+

Enhancing exceptions

+

Sometimes the BstError originates from a lower level component, +and the code segment which raised the exception did not have enough context +to create a complete, informative summary of the error for the user.

+

In these cases it is necessary to handle the error and raise a new +one, e.g.:

+
try:
+    extracted_artifact = self._artifacts.extract(self, cache_key)
+except ArtifactError as e:
+    raise ElementError("Failed to extract {} while checking out {}: {}"
+                       .format(cache_key, self.name, e)) from e
+
+
+
+
+

Programming errors

+

Sometimes you are writing code and have detected an unexpected condition, +or a broken invariant for which the code cannot be prepared to handle +gracefully.

+

In these cases, do not raise any of the BstError class exceptions.

+

Instead, use the assert statement, e.g.:

+
assert utils._is_main_process(), \
+    "Attempted to save workspace configuration from child process"
+
+
+

This will result in a BUG message with the stack trace included being +logged and reported in the frontend.

+
+
+

BstError parameters

+

When raising BstError class exceptions, there are some common properties +which can be useful to know about:

+
    +

  • message: The brief human readable error, will be formatted on one line in the frontend.

    • +

  • detail: An optional detailed human readable message to accompany the message summary +of the error. This is often used to recommend the user some course of action, or to provide +additional context about the error.

    • +

  • temporary: Some errors are allowed to be temporary, this attribute is only +observed from child processes which fail in a temporary way. This distinction +is used to determine whether the task should be retried or not. An error is usually +only a temporary error if the cause of the error was a network timeout.

    • +

  • reason: A machine readable identifier for the error. This is used for the purpose +of regression testing, such that we check that BuildStream has errored out for the +expected reason in a given failure mode.

    • +
+
+
+

Documenting Exceptions

+

We have already seen some examples of how +exceptions are documented in API documenting comments, but this is worth some +additional disambiguation.

+
    +

  • Only document the exceptions which are raised directly by the function in question. +It is otherwise nearly impossible to keep track of what exceptions might be raised +indirectly by calling the given function.

    • +

  • For a regular public or private method, your audience is a caller of the function; +document the exception in terms of what exception might be raised as a result of +calling this method.

    • +

  • For an abstract method, your audience is the +implementor of the method in a subclass; document the exception in terms of what +exception is prescribed for the implementing class to raise.

    • +
+
+
+
+

Always be consistent

+

There are various ways to define functions and classes in Python, +which has evolved with various features over time.

+

In BuildStream, we may not have leveraged all of the nice features +we could have, that is okay, and where it does not break API, we +can consider changing it.

+

Even if you know there is a better way to do a given thing in +Python when compared to the way we do it in BuildStream, do not do it.

+

Consistency of how we do things in the codebase is more important +than the actual way in which things are done, always.

+

Instead, if you like a certain Python feature and think the BuildStream +codebase should use it, then propose your change on the mailing list. Chances +are that we will reach agreement to use your preferred approach, and +in that case, it will be important to apply the change unilaterally +across the entire codebase, such that we continue to have a consistent +codebase.

+
+
+

Avoid tail calling

+

With the exception of tail calling with simple functions from +the standard Python library, such as splitting and joining lines +of text and encoding/decoding text; always avoid tail calling.

+

Good:

+
# Variables that we will need declared up top
+context = self._get_context()
+workspaces = context.get_workspaces()
+
+...
+
+# Saving the workspace configuration
+workspaces.save_config()
+
+
+

Bad:

+
# Saving the workspace configuration
+self._get_context().get_workspaces().save_config()
+
+
+

Acceptable:

+
# Decode the raw text loaded from a log file for display,
+# join them into a single utf-8 string and strip away any
+# trailing whitespace.
+return '\n'.join([line.decode('utf-8') for line in lines]).rstrip()
+
+
+

When you need to obtain a delegate object via an accessor function, +either do it at the beginning of the function, or at the beginning +of a code block within the function that will use that object.

+

There are several reasons for this convention:

+
    +

  • When observing a stack trace, it is always faster and easier to +determine what went wrong when all statements are on separate lines.

    • +

  • We always want individual lines to trace back to their origin as +much as possible for the purpose of tracing the history of code +with git blame.

    +

    One day, you might need the Context or Workspaces object +in the same function for another reason, at which point it will +be unacceptable to leave the existing line as written, because it +will introduce a redundant accessor to the same object, so the +line written as:

    +
    self._get_context().get_workspaces().save_config()
+
    +
    +

    Will have to change at that point, meaning we lose the valuable +information of which commit originally introduced this call +when running git blame.

    +
    • +

  • For similar reasons, we prefer delegate objects be accessed near +the beginning of a function or code block so that there is less +chance that this statement will have to move in the future, if +the same function or code block needs the delegate object for any +other reason.

    +

    Asides from this, code is generally more legible and uniform when +variables are declared at the beginning of function blocks.

    +
    • +
+
+
+

Vertical stacking of modules

+

For the sake of overall comprehensiveness of the BuildStream +architecture, it is important that we retain vertical stacking +order of the dependencies and knowledge of modules as much as +possible, and avoid any cyclic relationships in modules.

+

For instance, the Source objects are owned by Element +objects in the BuildStream data model, and as such the Element +will delegate some activities to the Source objects in its +possession. The Source objects should however never call functions +on the Element object, nor should the Source object itself +have any understanding of what an Element is.

+

If you are implementing a low level utility layer, for example +as a part of the YAML loading code layers, it can be tempting +to derive context from the higher levels of the codebase which use +these low level utilities, instead of defining properly stand alone +APIs for these utilities to work: Never do this.

+

Unfortunately, unlike other languages where include files play +a big part in ensuring that it is difficult to make a mess; Python, +allows you to just call methods on arbitrary objects passed through +a function call without having to import the module which defines +those methods - this leads to cyclic dependencies of modules quickly +if the developer does not take special care of ensuring this does not +happen.

+
+
+

Minimize arguments in methods

+

When creating an object, or adding a new API method to an existing +object, always strive to keep as much context as possible on the +object itself rather than expecting callers of the methods to provide +everything the method needs every time.

+

If the value or object that is needed in a function call is a constant +for the lifetime of the object which exposes the given method, then +that value or object should be passed in the constructor instead of +via a method call.

+
+
+

Minimize API surfaces

+

When creating an object, or adding new functionality in any way, +try to keep the number of public, outward facing +symbols to a minimum, this is important for both +internal and public, plugin facing API surfaces.

+

When anyone visits a file, there are two levels of comprehension:

+
    +

  • What do I need to know in order to use this object.

    • +

  • What do I need to know in order to modify this object.

    • +
+

For the former, we want the reader to understand with as little effort +as possible, what the public API contract is for a given object and consequently, +how it is expected to be used. This is also why we +order the symbols of a class in such a way +as to keep all outward facing public API surfaces at the top of the file, so that the +reader never needs to dig deep into the bottom of the file to find something they +might need to use.

+

For the latter, when it comes to having to modify the file or add functionality, +you want to retain as much freedom as possible to modify internals, while +being sure that nothing external will be affected by internal modifications. +Less client facing API means that you have less surrounding code to modify +when your API changes. Further, ensuring that there is minimal outward facing +API for any module minimizes the complexity for the developer working on +that module, by limiting the considerations needed regarding external side +effects of their modifications to the module.

+

When modifying a file, one should not have to understand or think too +much about external side effects, when the API surface of the file is +well documented and minimal.

+

When adding new API to a given object for a new purpose, consider whether +the new API is in any way redundant with other API (should this value now +go into the constructor, since we use it more than once? could this +value be passed along with another function, and the other function renamed, +to better suit the new purposes of this module/object?) and repurpose +the outward facing API of an object as a whole every time.

+
+
+

Avoid transient state on instances

+

At times, it can be tempting to store transient state that is +the result of one operation on an instance, only to be retrieved +later via an accessor function elsewhere.

+

As a basic rule of thumb, if the value is transient and just the +result of one operation, which needs to be observed directly after +by another code segment, then never store it on the instance.

+

BuildStream is complicated in the sense that it is multi processed +and it is not always obvious how to pass the transient state around +as a return value or a function parameter. Do not fall prey to this +obstacle and pollute object instances with transient state.

+

Instead, always refactor the surrounding code so that the value +is propagated to the desired end point via a well defined API, either +by adding new code paths or changing the design such that the +architecture continues to make sense.

+
+
+

Refactor the codebase as needed

+

Especially when implementing features, always move the BuildStream +codebase forward as a whole.

+

Taking a short cut is alright when prototyping, but circumventing +existing architecture and design to get a feature implemented without +re-designing the surrounding architecture to accommodate the new +feature instead, is never acceptable upstream.

+

For example, let’s say that you have to implement a feature and you’ve +successfully prototyped it, but it launches a Job directly from a +Queue implementation to get the feature to work, while the Scheduler +is normally responsible for dispatching Jobs for the elements on +a Queue. This means that you’ve proven that your feature can work, +and now it is time to start working on a patch for upstream.

+

Consider what the scenario is and why you are circumventing the design, +and then redesign the Scheduler and Queue objects to accommodate for +the new feature and condition under which you need to dispatch a Job, +or how you can give the Queue implementation the additional context it +needs.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/grpc_protocols.html b/hacking/grpc_protocols.html new file mode 100644 index 000000000..ac0bff7ed --- /dev/null +++ b/hacking/grpc_protocols.html @@ -0,0 +1,161 @@ + + + + + + + + + Generating protocol buffers — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Generating protocol buffers

+

BuildStream uses protobuf and gRPC for serialization and communication with +artifact cache servers. This requires .proto files and Python code +generated from the .proto files using protoc. All these files live in the +src/buildstream/_protos directory. The generated files are included in the +git repository to avoid depending on grpcio-tools for user installations.

+
+

Regenerating code

+

When .proto files are modified, the corresponding Python code needs to +be regenerated. As a prerequisite for code generation you need to install +grpcio-tools using pip or some other mechanism:

+
pip3 install --user grpcio-tools
+
+
+

To actually regenerate the code:

+
./setup.py build_grpc
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/making_releases.html b/hacking/making_releases.html new file mode 100644 index 000000000..204d07c12 --- /dev/null +++ b/hacking/making_releases.html @@ -0,0 +1,360 @@ + + + + + + + + + Making releases — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Making releases

+

This is a checklist of activities which must be observed when creating +BuildStream releases, it is important to keep this section up to date +whenever the release process changes.

+
+

Requirements

+

There are a couple of requirements and accounts required in order +to publish a release.

+
    +

  • Ability to send email to dev@buildstream.apache.org.

    • +

  • Shell account at master.gnome.org.

    • +

  • Access to the BuildStream project on PyPI

    • +

  • An email client which still knows how to send emails in plain text.

    • +
+
+
+

Pre-release changes

+

Before actually rolling the release, here is a list of changes which +might need to be done in preparation of the release.

+
    +

  • Ensure that the man pages are up to date

    +

    The man pages are committed to the repository because we are +currently unable to integrate this generation into the setuptools +build phase, as outlined in issue #8.

    +

    If any of the user facing CLI has changed, or if any of the +related docstrings have changed, then you should +regenerate the man pages and +add/commit the results before wrapping a release.

    +
    • +

  • Ensure the documentation session HTML is up to date

    +

    The session HTML files are committed to the repository for multiple +reasons, one of them being that the documentation must be buildable +from within a release build environment so that downstream distribution +packagers can easily create the docs package.

    +

    This is currently only needed for the first stable release +in a stable line of releases, after this point the API is frozen +and will not change for the remainder of the stable release lifetime, +so nothing interesting will have changed in these session files.

    +

    If regeneration is needed, follow the instructions above.

    +
    • +

  • Ensure the NEWS entry is up to date and ready

    +

    For a stable release where features have not been added, we +should at least add some entries about the issues which have +been fixed since the last stable release.

    +

    For development releases, it is worthwhile going over the +existing entries and ensuring all the major feature additions +are mentioned and there are no redundancies.

    +
    • +

  • Push pre-release changes

    +

    Now that any final pre-release changes to generated files or NEWS have +been made, push these directly to the upstream repository.

    +

    Do not sit around waiting for CI or approval, these superficial changes +do not affect CI and you are intended to push these changes directly +to the upstream repository.

    +
    • +
+
+
+

Release process

+
    +

  • Ensure that the latest commit is passing in CI

    +

    Of course, we do not release software which does not pass it’s own +tests.

    +
    • +

  • Get the list of contributors

    +

    The list of contributors for a given list is a list of +any contributors who have landed any patches since the +last release.

    +

    An easy way to get this list is to ask git to summarize +the authors of commits since the last release tag. For +example, if we are about to create the 1.1.1 release, then +we need to observe all of the commits since the 1.1.0 +release:

    +
    git shortlog -s 1.1.0...@
+
    +
    +

    At times, the same contributor might make contributions from different +machines which they have setup their author names differently, you +can see that some of the authors are actually duplicates, then +remove the duplicates.

    +
    • +

  • Start composing the release announcement email

    +

    The first thing to do when composing the release email is to +ensure your mail client has disabled any HTML formatting and will +safely use plain text only.

    +

    Try to make the release announcement consistent with other release +announcements as much as possible, an example of the email +can be found here.

    +

    The recipient of the email is dev@buildstream.apache.org and the title +of the email should be of the form: BuildStream 1.1.1 released, without +any exclamation point.

    +

    The format of the email is essentially:

    +
    Hi all,
+
+This is the personalized message written to you about this
+release.
+
+If this is an unstable release, this should include a warning
+to this effect and an invitation to users to please help us
+test this release.
+
+This is also a good place to highlight specific bug fixes which
+users may have been waiting for, or highlight a new feature we
+want users to try out.
+
+
+What is BuildStream ?
+=====================
+This is a concise blurb which describes BuildStream in a couple of
+sentences, and is taken from the the README.rst.
+
+The easiest thing is to just copy this over from the last release email.
+
+
+=================
+buildstream 1.1.1
+=================
+This section is directly copy pasted from the top of the NEWS file
+
+
+Contributors
+============
+ - This is Where
+ - You Put
+ - The Contributor
+ - Names Which
+ - You Extracted
+ - Using git shortlog -s
+
+
+Where can I get it ?
+====================
+https://download.gnome.org/sources/BuildStream/1.1/
+
+For more information on the BuildStream project, visit our home page
+at https://buildstream.build/
+
    +
    +
    • +

  • Publish the release tag

    +

    Now that any pre-release changes are upstream, create and push the +signed release tag like so:

    +
    git tag -s 1.1.1
+git push origin 1.1.1
+
    +
    +

    This will trigger the “Release actions” workflow which also takes care of:

    +
    +
      +

    • uploading Github release artifacts

      • +

    • uploading Python source and binary packages to PyPI

      • +
    +
    +
    • +

  • Upload the release tarball

    +

    First get yourself into a clean repository state, ensure that you +don’t have any unfinished work or precious, uncommitted files lying +around in your checkout and then run:

    +
    git clean -xdff
+
    +
    +

    Create the tarball with the following command:

    +
    python3 setup.py sdist
+
    +
    +

    And upload the resulting tarball to the master GNOME server:

    +
    scp dist/BuildStream-1.1.1.tar.gz <user>@master.gnome.org:
+
    +
    +

    And finally login to your account at master.gnome.org and run +the install scripts to publish the tarball and update the mirrors:

    +
    ftpadmin install BuildStream-1.1.1.tar.gz
+
    +
    +
    • +

  • Send the release email

    +

    Now that the release tag is up and the tarball is published, +you can send the release email.

    +
    • +
+
+
+

Post-release activities

+

Once the release has been published, there are some activities +which need to be done to ensure everything is up to date.

+
    +

  • Check that the release was successfully uploaded to PyPI. If +it is an unstable release, make sure the version on PyPI has +the correct “dev0” postfix so to avoid being treated as stable.

    • +

  • Update the topic line in the #buildstream IRC channel if needed

    +

    The IRC channel usually advertizes the latest stable release +in the topic line, now is the right time to update it.

    +
    • +

  • Update the website repository

    +

    The website wants to link to release announcements, but this +cannot be automated because we cannot guess what the link to +the release email will be in the mailing list archive.

    +

    Find the URL to the announcement you just published +in the mailing list archives, +and use that URL to update the anouncements.json file in the website +repository.

    +

    Commit and push this change to the the anouncements.json file to +the upstream website repository, and gitlab will take care of automatically +updating the website accordingly.

    +
    • +

  • Regenerate BuildStream documentation

    +

    In order to update the badges we use in various documentation +which reflects what is the latest stable releases and the latest +development snapshots, we simply need to ensure a pipeline runs +for the master branch in the BuildStream repository.

    +

    You can do this by using the “Run Pipeline” feature on the +pipelines page in the gitlab UI.

    +
    • +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/managing_data_files.html b/hacking/managing_data_files.html new file mode 100644 index 000000000..d1ae3bcc7 --- /dev/null +++ b/hacking/managing_data_files.html @@ -0,0 +1,148 @@ + + + + + + + + + Managing data files — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Managing data files

+

When adding data files which need to be discovered at runtime by BuildStream, update setup.py accordingly.

+

When adding data files for the purpose of docs or tests, or anything that is not covered by +setup.py, update the MANIFEST.in accordingly.

+

At any time, running the following command to create a source distribution should result in +creating a tarball which contains everything we want it to include:

+
./setup.py sdist
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/measuring_performance.html b/hacking/measuring_performance.html new file mode 100644 index 000000000..9be971160 --- /dev/null +++ b/hacking/measuring_performance.html @@ -0,0 +1,226 @@ + + + + + + + + + Measuring performance — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Measuring performance

+
+

Benchmarking framework

+

BuildStream has a utility to measure performance which is available from a +separate repository at https://gitlab.com/BuildStream/benchmarks. This tool +allows you to run a fixed set of workloads with multiple versions of +BuildStream. From this you can see whether one version performs better or +worse than another which is useful when looking for regressions and when +testing potential optimizations.

+

For full documentation on how to use the benchmarking tool see the README in +the ‘benchmarks’ repository.

+
+
+

Profiling tools

+

When looking for ways to speed up the code you should make use of a profiling +tool.

+

Python provides cProfile +which gives you a list of all functions called during execution and how much +time was spent in each function. Here is an example of running bst --help +under cProfile:

+
+

python3 -m cProfile -o bst.cprofile – $(which bst) –help

+
+

You can then analyze the results interactively using the ‘pstats’ module:

+
+

python3 -m pstats ./bst.cprofile

+
+

For more detailed documentation of cProfile and ‘pstats’, see: +https://docs.python.org/3/library/profile.html.

+

For a richer and interactive visualisation of the .cprofile files, you can +try snakeviz. +You can install it with pip install snakeviz. Here is an example invocation:

+
+

snakeviz bst.cprofile

+
+

It will then start a webserver and launch a browser to the relevant page.

+
+

Note

+

If you want to also profile cython files, you will need to add +BST_CYTHON_PROFILE=1 and recompile the cython files. +pip install would take care of that.

+
+
+
+

Profiling specific parts of BuildStream with BST_PROFILE

+

BuildStream can also turn on cProfile for specific parts of execution +using BST_PROFILE.

+

BST_PROFILE can be set to a section name, or a list of section names separated +by “:”. You can also use “all” for getting all profiles at the same time. +There is a list of topics in src/buildstream/_profile.py. For example, running:

+
BST_PROFILE=load-pipeline bst build bootstrap-system-x86.bst
+
+
+

will produce a profile in the current directory for the time take to +call most of initialized, for each element. These profile files +are in the same cProfile format as those mentioned in the previous +section, and can be analysed in the same way.

+
+
+

Fixing performance issues

+

BuildStream uses Cython in order to speed up specific parts of the +code base.

+
+

Note

+

When optimizing for performance, please ensure that you optimize the algorithms before +jumping into Cython code. Cython will make the code harder to maintain and less accessible +to all developers.

+
+

When adding a new cython file to the codebase, you will need to register it in setup.py.

+

For example, for a module buildstream._my_module, to be written in src/buildstream/_my_module.pyx, you would do:

+
register_cython_module("buildstream._my_module")
+
+
+

In setup.py and the build tool will automatically use your module.

+
+

Note

+

Please register cython modules at the same place as the others.

+
+

When adding a definition class to share cython symbols between modules, please document the implementation file +and only expose the required definitions.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/ui.html b/hacking/ui.html new file mode 100644 index 000000000..bb05120cd --- /dev/null +++ b/hacking/ui.html @@ -0,0 +1,209 @@ + + + + + + + + + Contributing to the UI — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Contributing to the UI

+

As we wish to cleanly separate BuildStream’s core from the frontend, anything +user facing should be defined within the modules contained within the _frontend +directory.

+

BuildStream’s frontend includes:

+
+
    +

  • Implementation of the command line interface (cli.py)

    • +

  • Logline/text formatting (widget.py)

    • +

  • The main application state (app.py) which initializes the stream to handle +logging and user interactions.

    • +

  • Colour profiles (profile.py)

    • +

  • Rendering of the status bar (status.py)

    • +

  • Autocompletion behaviour (completions.py)

    • +
+
+
+

The command line interface

+

All of BuildStream’s commands are defined within the module cli.py -, +the main entry point which implements the CLI.

+

The command line interface is generated with Click - a third party Python package. Click +is easy to use and automatically generates help pages.

+

When working with commands, please adhere to the following checklist:

+
    +

  1. All commands should be defined with a help text

    2. +

  2. The command should be placed within the appropriate sub group

    + +
    3. +

  3. If the command is intended to work with artifact refs as well as element +names, the command’s argument should be “artifacts” as this supports the +auto-completion of artifact refs.

    4. +

  4. The supported –deps options are: “run”, “build”, “all”, “plan” and “none”. +These are always of type click.Choice and +should always be specified with a default. If the default is “none”, note that +we use the string “none”, not the Python built-in None. In addition to this, +the show_default flag should be set to True.

    5. +

  5. Commands should use the app and go through the stream (via a similarly named +method within Stream) in order to communicate to BuildStream’s core.

    6. +
+
+
+

Displaying information

+

Output which we wish to display to the user from the frontend should use the +implemented classes in widget.py. This module contains classes which represent +widgets for displaying information to the user.

+

To report messages back to the frontend, we use the Message() object +which is available from the Context.

+

Supported message types are defined in _message.py +and various uses of the messenger are defined in _messenger.py

+

The Messenger class defines various methods which allow us to report back to +the frontend in particular ways. The common methods are:

+
    +

  • Messenger.message() - the central point through which all messages pass

    • +

  • Messenger.timed_activity() - a context manager for performing and logging +timed activities.

    • +

  • Messenger.simple_task() - a Context manager for creating a task to report +progress too.

    • +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/updating_python_deps.html b/hacking/updating_python_deps.html new file mode 100644 index 000000000..fa9a532e1 --- /dev/null +++ b/hacking/updating_python_deps.html @@ -0,0 +1,250 @@ + + + + + + + + + Updating BuildStream’s Python dependencies — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Updating BuildStream’s Python dependencies

+

BuildStream’s Python dependencies are listed in multiple +requirements files +present in the requirements directory.

+

All .txt files in this directory are generated from the corresponding +.in file, and each .in file represents a set of dependencies. For +example, requirements.in contains all runtime dependencies of BuildStream. +requirements.txt is generated from it, and contains pinned versions of all +runtime dependencies (including transitive dependencies) of BuildStream.

+

When adding a new dependency to BuildStream, or updating existing dependencies, +it is important to update the appropriate requirements file accordingly. After +changing the .in file, run the following to update the matching .txt +file:

+
make -C requirements
+
+
+
+
+

Adding support for a new Python release

+

When a new stable release of Python 3 appears, we must explicitly declare +our support for it in the following places.

+
+

tox.ini

+

The tox.ini file defines the environments where the BuildStream test suite +runs. Every py{3.x,3.y} list must be updated to contain the new version +number such as 311 for CPython 3.11.

+

Use tox -e py311-nocover to run the test suite with the new version of +Python.

+
+
+

pyproject.toml

+
+

Bump cython version

+

New releases of Cython must be depended on with new versions of Python +in lock step.

+

When supporting a new Python version, it is important to bump the minimal +dependency on Cython to a new enough version which also supports the new +version of Python.

+
+
+

Wheel details

+

We produce binary “wheel” packages for each supported version of Python. +The cibuildwheel build tool will build for all released versions of Python +so no change is needed in the config.

+

However, if you want to test wheel building with a prerelease version of Python +you will need to set CIBW_PRERELEASE_PYTHONS=1 in the cibuildwheel +environment.

+
+
+
+

.github/compose/ci.docker-compose.yml

+

Each binary package is tested in a container, using the +pypa/manylinux images.

+

You need to add a new docker-compose service here – copy the +latest one and update the version number where it appears.

+
+
+

.github/workflows/ci.yml and .github/workflows/release.yml

+

There is a separate CI job to run each of the above tests. Update the +matrix config for the test_wheels jobs in ci.yml and release.yml +to add the new Python version.

+
+
+
+

Removing support for a Python release

+
+

tox.ini

+

You will need to update the py{3.x,3.y} lists to remove the old version. In +the envlist section, make sure the oldest version still has coverage +enabled while the other versions are marked -nocover.

+
+
+

pyproject.toml

+

The cibuildwheel tool will produce wheels for all versions of Python supported +upstream.. If we drop support for a version before upstream do, update the +tool.cibuildwheel.skip list to skip all platform tags for that version. +The glob cp36-* would skip all CPython 3.6 builds, for example.

+
+
+

.github/compose/ci.docker-compose.yml

+

Remove the corresponding service.

+
+
+

.github/workflows/ci.yml and .github/workflows/release.yml

+

Update the matrix config for the test_wheels jobs in ci.yml and +release.yml to remove the old Python version.

+
+
+
+

ABI compatibility for binary Python packages

+

The Python binary packages declare system requirements using +platform compatibility tags.

+

For linux-gnu systems we use manylinux_x_y platform tags +to specify a minimum GLIBC version. The platform tag is controlled in pyproject.toml with the +tool.cibuildwheel.manylinux-x86_64-image key. It must correspond with the version of +GLIBC used in buildbox-integration +to produce static buildbox binaries that are included in the package. +The cibuildwheel tool uses auditwheel +to ensure the correct platform tag is declared.

+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/using_the_testsuite.html b/hacking/using_the_testsuite.html new file mode 100644 index 000000000..bc283aee6 --- /dev/null +++ b/hacking/using_the_testsuite.html @@ -0,0 +1,387 @@ + + + + + + + + + Using the test suite — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Using the test suite

+

BuildStream uses tox as a frontend to run the +tests which are implemented using pytest. We use +pytest for regression tests and testing out the behavior of newly added +components.

+

The elaborate documentation for pytest can be found here: http://doc.pytest.org/en/latest/contents.html

+

Don’t get lost in the docs if you don’t need to, follow existing examples instead.

+
+

Installing build dependencies

+

Some of BuildStream’s dependencies have non-python build dependencies. When +running tests with tox, you will first need to install these dependencies. +Exact steps to install these will depend on your operating system. Commands +for installing them for some common distributions are listed below.

+

For Fedora-based systems:

+
dnf install gcc python3-devel
+
+
+

For Debian-based systems:

+
apt install gcc python3-dev
+
+
+
+
+

Installing runtime dependencies

+

To be able to run BuildStream as part of the test suite, BuildStream’s runtime +dependencies must also be installed. Instructions on how to do so can be found +in Installing Dependencies.

+

If you are not interested in running the integration tests, you can skip the +installation of buildbox-run.

+
+
+

Running tests

+

To run the tests, simply navigate to the toplevel directory of your BuildStream +checkout and run:

+
tox
+
+
+

By default, the test suite will be run against every supported python version +found on your host. If you have multiple python versions installed, you may +want to run tests against only one version and you can do that using the -e +option when running tox:

+
tox -e py37
+
+
+

If you would like to test and lint at the same time, or if you do have multiple +python versions installed and would like to test against multiple versions, then +we recommend using detox, just run it with +the same arguments you would give tox:

+
detox -e lint,py36,py37
+
+
+

The output of all failing tests will always be printed in the summary, but +if you want to observe the stdout and stderr generated by a passing test, +you can pass the -s option to pytest as such:

+
tox -- -s
+
+
+
+

Tip

+

The -s option is a pytest option.

+

Any options specified before the -- separator are consumed by tox, +and any options after the -- separator will be passed along to pytest.

+
+

You can always abort on the first failure by running:

+
tox -- -x
+
+
+

Similarly, you may also be interested in the --last-failed and +--failed-first options as per the +pytest cache documentation.

+

If you want to run a specific test or a group of tests, you +can specify a prefix to match. E.g. if you want to run all of +the frontend tests you can do:

+
tox -- tests/frontend/
+
+
+

Specific tests can be chosen by using the :: delimiter after the test module. +If you wanted to run the test_build_track test within frontend/buildtrack.py you could do:

+
tox -- tests/frontend/buildtrack.py::test_build_track
+
+
+

When running only a few tests, you may find the coverage and timing output +excessive, there are options to trim them. Note that coverage step will fail. +Here is an example:

+
tox -- --no-cov --durations=1 tests/frontend/buildtrack.py::test_build_track
+
+
+

We also have a set of slow integration tests that are disabled by +default - you will notice most of them marked with SKIP in the pytest +output. To run them, you can use:

+
tox -- --integration
+
+
+

In case BuildStream’s dependencies were updated since you last ran the +tests, you might see some errors like +pytest: error: unrecognized arguments: --codestyle. If this happens, you +will need to force tox to recreate the test environment(s). To do so, you +can run tox with -r or --recreate option.

+
+

Note

+

By default, we do not allow use of site packages in our tox +configuration to enable running the tests in an isolated environment. +If you need to enable use of site packages for whatever reason, you can +do so by passing the --sitepackages option to tox. Also, you will +not need to install any of the build dependencies mentioned above if you +use this approach.

+
+
+

Note

+

While using tox is practical for developers running tests in +more predictable execution environments, it is still possible to +execute the test suite against a specific installation environment +using pytest directly:

+
pytest
+
+
+

If you want to run coverage, you will need need to add BST_CYTHON_TRACE=1 +to your environment if you also want coverage on cython files. You could then +get coverage by running:

+
BST_CYTHON_TRACE=1 coverage run pytest
+
+
+

Note that you will have to have all dependencies installed already, when +running tests directly via pytest. This includes the following:

+
    +

  • Cython and Setuptools, as build dependencies

    • +

  • Runtime dependencies and test dependencies are specified in requirements +files, present in the requirements subdirectory. Refer to the .in +files for loose dependencies and .txt files for fixed version of all +dependencies that are known to work.

    • +

  • Additionally, if you are running tests that involve external plugins, you +will need to have those installed as well.

    • +
+

You can also have a look at our tox configuration in tox.ini file if you +are unsure about dependencies.

+
+
+

Tip

+

We also have an environment called ‘venv’ which takes any arguments +you give it and runs them inside the same virtualenv we use for our +tests:

+
tox -e venv -- <your command(s) here>
+
+
+

Any commands after -- will be run a virtualenv managed by tox.

+
+
+
+

Running linters

+

Linting is performed separately from testing. In order to run the linting step which +consists of running the pylint tool, run the following:

+
tox -e lint
+
+
+
+

Tip

+

The project specific pylint configuration is stored in the toplevel +buildstream directory in the .pylintrc file. This configuration can be +interesting to use with IDEs and other developer tooling.

+
+
+
+

Formatting code

+

Similar to linting, code formatting is also done via a tox environment. To +format the code using the black tool, run the following:

+
tox -e format
+
+
+
+
+

Observing coverage

+

Once you have run the tests using tox (or detox), some coverage reports will +have been left behind.

+

To view the coverage report of the last test run, simply run:

+
tox -e coverage
+
+
+

This will collate any reports from separate python environments that may be +under test before displaying the combined coverage.

+
+
+

Adding tests

+

Tests are found in the tests subdirectory, inside of which +there is a separate directory for each domain of tests. +All tests are collected as:

+
tests/*/*.py
+
+
+

If the new test is not appropriate for the existing test domains, +then simply create a new directory for it under the tests subdirectory.

+

Various tests may include data files to test on, there are examples +of this in the existing tests. When adding data for a test, create +a subdirectory beside your test in which to store data.

+

When creating a test that needs data, use the datafiles extension +to decorate your test case (again, examples exist in the existing +tests for this), documentation on the datafiles extension can +be found here: https://pypi.python.org/pypi/pytest-datafiles.

+

Tests that run a sandbox should be decorated with:

+
@pytest.mark.integration
+
+
+

and use the integration cli helper.

+

You must test your changes in an end-to-end fashion. Consider the first end to +be the appropriate user interface, and the other end to be the change you have +made.

+

The aim for our tests is to make assertions about how you impact and define the +outward user experience. You should be able to exercise all code paths via the +user interface, just as one can test the strength of rivets by sailing dozens +of ocean liners. Keep in mind that your ocean liners could be sailing properly +because of a malfunctioning rivet. End-to-end testing will warn you that +fixing the rivet will sink the ships.

+

The primary user interface is the cli, so that should be the first target ‘end’ +for testing. Most of the value of BuildStream comes from what you can achieve +with the cli.

+

We also have what we call a “Public API Surface”, as previously mentioned in +Documenting symbols. You should consider this a secondary +target. This is mainly for advanced users to implement their plugins against.

+

Note that both of these targets for testing are guaranteed to continue working +in the same way across versions. This means that tests written in terms of them +will be robust to large changes to the code. This important property means that +BuildStream developers can make large refactorings without needing to rewrite +fragile tests.

+

Another user to consider is the BuildStream developer, therefore internal API +surfaces are also targets for testing. For example the YAML loading code, and +the CasCache. Remember that these surfaces are still just a means to the end of +providing value through the cli and the “Public API Surface”.

+

It may be impractical to sufficiently examine some changes in an end-to-end +fashion. The number of cases to test, and the running time of each test, may be +too high. Such typically low-level things, e.g. parsers, may also be tested +with unit tests; alongside the mandatory end-to-end tests.

+

It is important to write unit tests that are not fragile, i.e. in such a way +that they do not break due to changes unrelated to what they are meant to test. +For example, if the test relies on a lot of BuildStream internals, a large +refactoring will likely require the test to be rewritten. Pure functions that +only rely on the Python Standard Library are excellent candidates for unit +testing.

+

Unit tests only make it easier to implement things correctly, end-to-end tests +make it easier to implement the right thing.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/writing_documentation.html b/hacking/writing_documentation.html new file mode 100644 index 000000000..dd099ca93 --- /dev/null +++ b/hacking/writing_documentation.html @@ -0,0 +1,392 @@ + + + + + + + + + Writing documentation — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Writing documentation

+

BuildStream starts out as a documented project from day one and uses +sphinx to document itself.

+

This section discusses formatting policies for editing files in the +doc/source directory, and describes the details of how the docs are +generated so that you can easily generate and view the docs yourself before +submitting patches to the documentation.

+

For details on how API documenting comments and docstrings are formatted, +refer to the documenting section of the coding guidelines.

+
+

Documentation formatting policy

+

The BuildStream documentation style is as follows:

+
    +

  • Titles and headings require two leading empty lines above them. +Only the first word in a title should be capitalized.

    +
      +

    • If there is an .. _internal_link: anchor, there should be two empty lines +above the anchor, followed by one leading empty line.

      • +
    +
    • +

  • Within a section, paragraphs should be separated by one empty line.

    • +

  • Notes are defined using: .. note:: blocks, followed by an empty line +and then indented (3 spaces) text.

    +
      +

    • Other kinds of notes can be used throughout the documentation and will +be decorated in different ways, these work in the same way as .. note:: does.

      +

      Feel free to also use .. attention:: or .. important:: to call special +attention to a paragraph, .. tip:: to give the reader a special tip on how +to use an advanced feature or .. warning:: to warn the user about a potential +misuse of the API and explain its consequences.

      +
      • +
    +
    • +

  • Code blocks are defined using: .. code:: LANGUAGE blocks, followed by an empty +line and then indented (3 spaces) text. Note that the default language is python.

    • +

  • Cross references should be of the form :role:`target`.

    +
      +

    • Explicit anchors can be declared as .. _anchor_name: on a line by itself.

      • +

    • To cross reference arbitrary locations with, for example, the anchor anchor_name, +always provide some explicit text in the link instead of deriving the text from +the target, e.g.: :ref:`Link text <anchor_name>`. +Note that the “_” prefix is not used when referring to the target.

      • +
    +
    • +
+

For further information about using the reStructuredText with sphinx, please see the +Sphinx Documentation.

+
+
+

Building Docs

+

Before you can build the docs, you will end to ensure that you have installed +the required build dependencies as mentioned +in the testing section above.

+

To build the documentation, just run the following:

+
tox -e docs
+
+
+

This will give you a doc/build/html directory with the html docs which +you can view in your browser locally to test.

+
+

Regenerating session html

+

The documentation build will build the session files if they are missing, +or if explicitly asked to rebuild. We revision the generated session html files +in order to reduce the burden on documentation contributors.

+

To explicitly rebuild the session snapshot html files, it is recommended that you +first set the BST_SOURCE_CACHE environment variable to your source cache, this +will make the docs build reuse already downloaded sources:

+
export BST_SOURCE_CACHE=~/.cache/buildstream/sources
+
+
+

To force rebuild session html while building the doc, simply run tox with the +BST_FORCE_SESSION_REBUILD environment variable set, like so:

+
env BST_FORCE_SESSION_REBUILD=1 tox -e docs
+
+
+
+
+
+

Man pages

+

Unfortunately it is quite difficult to integrate the man pages build +into the setup.py, as such, whenever the frontend command line +interface changes, the static man pages should be regenerated and +committed with that.

+

To do this, run the following from the the toplevel directory of BuildStream:

+
tox -e man
+
+
+

And commit the result, ensuring that you have added anything in +the man/ subdirectory, which will be automatically included +in the buildstream distribution.

+
+
+

User guide

+

The user guide is comprised of free form documentation +in manually written .rst files and is split up into a few sections, +of main interest are the tutorial and the +examples.

+

The distinction of the two categories of user guides is important to +understand too.

+
    +

  • Tutorial

    +

    The tutorial is structured as a series of exercises which start with +the most basic concepts and build upon the previous chapters in order +to arrive at a basic understanding of how to create BuildStream projects.

    +

    This series of examples should be easy enough to complete in a matter +of a few hours for a new user, and should provide just enough insight to +get the user started in creating their own projects.

    +

    Going through the tutorial step by step should also result in the user +becoming proficient enough with the reference manual to get by on their own.

    +
    • +

  • Examples

    +

    These exist to demonstrate how to accomplish more advanced tasks which +are not always obvious and discoverable.

    +

    Alternatively, these also demonstrate elegant and recommended ways of +accomplishing some tasks which could be done in various ways.

    +
    • +
+
+

Guidelines

+

Here are some general guidelines for adding new free form documentation +to the user guide.

+
    +

  • Focus on a single subject

    +

    It is important to stay focused on a single subject and avoid getting +into tangential material when creating a new entry, so that the articles +remain concise and the user is not distracted by unrelated subject material.

    +

    A single tutorial chapter or example should not introduce any additional +subject material than the material being added for the given example.

    +
    • +

  • Reuse existing sample project elements

    +

    To help avoid distracting from the topic at hand, it is always preferable to +reuse the same project sample material from other examples and only deviate +slightly to demonstrate the new material, than to create completely new projects.

    +

    This helps us remain focused on a single topic at a time, and reduces the amount +of unrelated material the reader needs to learn in order to digest the new +example.

    +
    • +

  • Don’t be redundant

    +

    When something has already been explained in the tutorial or in another example, +it is best to simply refer to the other user guide entry in a new example.

    +

    Always prefer to link to the tutorial if an explanation exists in the tutorial, +rather than linking to another example, where possible.

    +
    • +

  • Link into the reference manual at every opportunity

    +

    The format and plugin API is 100% documented at all times. Whenever discussing +anything about the format or plugin API, always do so while providing a link +into the more terse reference material.

    +

    We don’t want users to have to search for the material themselves, and we also +want the user to become proficient at navigating the reference material over +time.

    +
    • +

  • Use concise terminology

    +

    As developers, we tend to come up with code names for features we develop, and +then end up documenting a new feature in an example.

    +

    Never use a code name or shorthand to refer to a feature in the user guide, instead +always use fully qualified sentences outlining very explicitly what we are doing +in the example, or what the example is for in the case of a title.

    +

    We need to be considerate that the audience of our user guide is probably a +proficient developer or integrator, but has no idea what we might have decided +to name a given activity.

    +
    • +
+
+
+

Structure of an example

+

The tutorial and the examples sections +of the documentation contain a series of sample projects, each chapter in +the tutorial, or standalone example uses a sample project.

+

Here is the the structure for adding new examples and tutorial chapters.

+
    +

  • The example has a ${name}.

    • +

  • The example has a project users can copy and use.

    +
      +

    • This project is added in the directory doc/examples/${name}.

      • +
    +
    • +

  • The example has a documentation component.

    +
      +

    • This is added at doc/source/examples/${name}.rst

      • +

    • An entry for examples/${name} is added to the toctree in doc/source/using_examples.rst

      • +

    • This documentation discusses the project elements declared in the project and may +provide some BuildStream command examples.

      • +

    • This documentation links out to the reference manual at every opportunity.

      • +
    +
    +

    Note

    +

    In the case of a tutorial chapter, the .rst file is added in at +doc/source/tutorial/${name}.rst and an entry for tutorial/${name} +is added to doc/source/using_tutorial.rst.

    +
    +
    • +

  • The example has a CI test component.

    +
      +

    • This is an integration test added at tests/examples/${name}.

      • +

    • This test runs BuildStream in the ways described in the example +and assert that we get the results which we advertize to users in +the said examples.

      • +
    +
    • +
+
+
+
+

Adding BuildStream command output

+

As a part of building the docs, BuildStream will run itself and extract +some html for the colorized output which is produced.

+

If you want to run BuildStream to produce some nice html for your +documentation, then you can do so by adding new .run files to the +doc/sessions/ directory.

+

Any files added as doc/sessions/${example}.run will result in generated +file at doc/source/sessions/${example}.html, and these files can be +included in the reStructuredText documentation at any time with:

+
.. raw:: html
+   :file: sessions/${example}.html
+
+
+

The .run file format is just another YAML dictionary which consists of a +commands list, instructing the program what to do command by command.

+

Each command is a dictionary, the members of which are listed here:

+
    +

  • directory: The input file relative project directory.

    • +

  • output: The input file relative output html file to generate (optional).

    • +

  • fake-output: Don’t really run the command, just pretend to and pretend +this was the output, an empty string will enable this too.

    • +

  • command: The command to run, without the leading bst.

    • +

  • shell: Specifying True indicates that command should be run as +a shell command from the project directory, instead of a bst command (optional).

    • +
+

When adding a new .run file, one should normally also commit the new +resulting generated .html file(s) into the doc/source/sessions-stored/ +directory at the same time, this ensures that other developers do not need to +regenerate them locally in order to build the docs.

+

Example:

+
commands:
+
+# Make it fetch first
+- directory: ../examples/foo
+  command: source fetch hello.bst
+
+# Capture a build output
+- directory: ../examples/foo
+  output: ../source/sessions/foo-build.html
+  command: build hello.bst
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/hacking/writing_plugins.html b/hacking/writing_plugins.html new file mode 100644 index 000000000..e7172caf2 --- /dev/null +++ b/hacking/writing_plugins.html @@ -0,0 +1,184 @@ + + + + + + + + + Adding core plugins — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Adding core plugins

+

This is a checklist of things which need to be done when adding a new +core plugin to BuildStream proper.

+
+

Update documentation index

+

The documentation generating scripts will automatically pick up your +newly added plugin and generate HTML, but will not add a link to the +documentation of your plugin automatically.

+

Whenever adding a new plugin, you must add an entry for it in doc/source/core_plugins.rst.

+
+
+

Add tests

+

Needless to say, all new feature additions need to be tested. For Element +plugins, these usually need to be added to the integration tests. For Source +plugins, the tests are added in two ways:

+
    +

  • For most normal Source plugins, it is important to add a new Repo +implementation for your plugin in the tests/testutils/repo/ directory +and update ALL_REPO_KINDS in tests/testutils/repo/__init__.py. This +will include your new Source implementation in a series of already existing +tests, ensuring it works well under normal operating conditions.

    • +

  • For other source plugins, or in order to test edge cases, such as failure modes, +which are not tested under the normal test battery, add new tests in tests/sources.

    • +
+
+
+

Extend the cachekey test

+

For any newly added plugins, it is important to add some new simple elements +in tests/cachekey/project/elements or tests/cachekey/project/sources, +and ensure that the newly added elements are depended on by tests/cachekey/project/target.bst.

+

One new element should be added to the cache key test for every configuration +value which your plugin understands which can possibly affect the result of +your plugin’s Plugin.get_unique_key() implementation.

+

This test ensures that cache keys do not unexpectedly change or become incompatible +due to code changes. As such, the cache key test should have full coverage of every +YAML configuration which can possibly affect cache key outcome at all times.

+

See the src/buildstream/testing/_update_cachekeys.py file for instructions on running the updater, +you need to run the updater to generate the .expected files and add the new +.expected files in the same commit which extends the cache key test.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/handling-files/composition.html b/handling-files/composition.html new file mode 100644 index 000000000..e3d0a43f3 --- /dev/null +++ b/handling-files/composition.html @@ -0,0 +1,814 @@ + + + + + + + + + 1. Composition — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

1. Composition

+

In this chapter we will explore how to create compositions of multiple +input filesystem trees, using the compose element.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/composition +subdirectory.

+
+
+

1.1. Overview

+

Composing a directory tree based on a set of build dependencies is often +one of the important steps you might perform in order to create a single artifact +which can be checked out and deployed.

+

In order to use the compose element, it is important +to first understand the concept of split rules, which +we will cover in this chapter.

+
+

1.1.1. Introducing split rules

+

The split rules of an element declaration denote +which sets of files in the given element’s resulting artifact belong to which +domain name.

+

The domains can then be used in various ways, using plugins which understand +split rule domains.

+

BuildStream’s default project configuration +contains a sensible set of default split rule domains for the purpose of +artifact splitting, they can be overridden in your project.conf, +and finally on a per element basis in the public data +of your element declarations.

+
+

Note

+

Projects are free to add additional split rule domains on top of the +default domains provided by the default project configuration.

+

There is nothing wrong with defining split rule domains which overlap, +possibly capturing some of the same files also captured by another +domain, however you should be aware of this when later using your +split rules with a plugin which processes them, like the +compose element described in this chapter.

+
+
+

1.1.1.1. Example of split rule declaration

+

In an element, you might need to define or extend the split-rules +in order to capture files in custom locations in a logical domain.

+

Here is an example of how you might use the +list append directive +to append an additional rule to your split-rules list in order to +capture additional data files which your application or library might +want to include in the runtime domain:

+
# Add our .dat files to the runtime domain
+public:
+  bst:
+    split-rules:
+      runtime:
+        (>):
+        - |
+          %{datadir}/foo/*.dat
+
+
+

Split rules are absolute paths which denote files within an artifact’s root +directory. The globbing patterns supported in split rules are defined in the +reference documentation here.

+
+

Important

+

Note that because of variable expansion, split rules can often be +resolved differently for elements which have overridden path +related variables, like %{prefix}.

+

This usually means that you do not need to explicitly extend or override +split rules on a specific element unless your element installs files to +special case locations.

+
+
+
+
+
+

1.2. Project structure

+

In this example we expand on the chapter about +integration commands, so we will +only discuss the files which are added or changed from that example.

+
+

1.2.1. elements/base/alpine.bst

+
kind: import
+description: |
+
+    Alpine Linux base runtime
+
+sources:
+- kind: tar
+  url: alpine:integration-tests-base.v1.x86_64.tar.xz
+  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639
+
+public:
+  bst:
+    #
+    # Run ldconfig in the libdir before running anything
+    #
+    integration-commands:
+    - ldconfig "%{libdir}"
+
+    #
+    # Extend the runtime split-rule domain for this element,
+    # such that we capture the runtime linker.
+    #
+    # There are various other things provided by this runtime
+    # such as tooling in /bin and an installation of python
+    # and perl, but we'll overlook these for the sake of
+    # this example.
+    #
+    split-rules:
+      runtime:
+        (>):
+        - "/lib/ld*.so*"
+
+
+

Here we have modified the base runtime, so as to specify that for this +element, we want to also include the runtime linker into the runtime domain.

+
+
+

1.2.2. elements/runtime-only.bst

+
kind: compose
+
+# Dependencies of a compose element cannot be transient,
+# we can only build-depend on the inputs of a composition.
+#
+build-depends:
+- hello.bst
+
+config:
+
+  # Only include files from the runtime domain
+  #
+  include:
+  - runtime
+
+  # Don't include any files which do not match any existing
+  # split rule domains.
+  #
+  include-orphans: False
+
+  # Run integration commands before composition
+  #
+  integrate: True
+
+
+

As we can see, this compose element has +been configured to only include files from the runtime domain.

+
+
+
+

1.3. Using the project

+

Now that we’ve presented how split rules +work and shown how to use them in the context of this example, lets +use the compose element we’ve created and +observe the results.

+
+

1.3.1. Building the project

+ +
+user@host:~/composition$ bst build runtime-only.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888154.665347    1483 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:15:54
+    Project:       composition (/home/user/buildstream/doc/examples/composition)
+    Targets:       runtime-only.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-_agmhsju/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-_agmhsju
+    Log Files:               /home/user/buildstream/doc/run-bst-_agmhsju/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-_agmhsju/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-_agmhsju/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: composition
+
+    Element Plugins
+        compose: core plugin
+        manual:  core plugin
+        stack:   core plugin
+        import:  core plugin
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Pipeline
+fetch needed 2f643c2da05dd1f300aa1b5aa016a925a6fecfdb3c9ff947c66a97002496aac2 base/alpine.bst 
+     waiting 82d7d3429a94cd01054753bbdc565d3f37069aacba51051764ec0f72129349f7 base.bst 
+     waiting 0babd9cc5beb41704ffe9a110ed8b4ca69ff8241a630b46176052544c07a35b2 libhello.bst 
+     waiting 6fb0f24a1fee36ba2f486fa0cd0cbedeb8f7a69df15908b9370aaa7799f8353d hello.bst 
+     waiting c22b89ed1a1f72aac4ee28fe220f9efa2c2d4002a1b1c414e659cfb089435960 runtime-only.bst 
+===============================================================================
+[--:--:--][2f643c2d][   fetch:base/alpine.bst               ] START   composition/base-alpine/2f643c2d-fetch.20241106-101554.log
+[--:--:--][0babd9cc][   fetch:libhello.bst                  ] START   composition/libhello/0babd9cc-fetch.20241106-101554.log
+[--:--:--][82d7d342][   fetch:base.bst                      ] START   composition/base/82d7d342-fetch.20241106-101554.log
+[--:--:--][c22b89ed][   fetch:runtime-only.bst              ] START   composition/runtime-only/c22b89ed-fetch.20241106-101554.log
+[--:--:--][2f643c2d][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:00][82d7d342][   fetch:base.bst                      ] SUCCESS composition/base/82d7d342-fetch.20241106-101554.log
+[00:00:00][c22b89ed][   fetch:runtime-only.bst              ] SUCCESS composition/runtime-only/c22b89ed-fetch.20241106-101554.log
+[00:00:00][0babd9cc][   fetch:libhello.bst                  ] SUCCESS composition/libhello/0babd9cc-fetch.20241106-101554.log
+[00:00:00][2f643c2d][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:06][2f643c2d][   fetch:base/alpine.bst               ] SUCCESS composition/base-alpine/2f643c2d-fetch.20241106-101554.log
+[--:--:--][2f643c2d][   build:base/alpine.bst               ] START   composition/base-alpine/2f643c2d-build.20241106-101601.log
+[--:--:--][2f643c2d][   build:base/alpine.bst               ] START   Staging sources
+[00:00:00][2f643c2d][   build:base/alpine.bst               ] SUCCESS Staging sources
+[--:--:--][2f643c2d][   build:base/alpine.bst               ] START   Caching artifact
+[00:00:00][2f643c2d][   build:base/alpine.bst               ] SUCCESS Caching artifact
+[00:00:00][2f643c2d][   build:base/alpine.bst               ] SUCCESS composition/base-alpine/2f643c2d-build.20241106-101601.log
+[--:--:--][82d7d342][   build:base.bst                      ] START   composition/base/82d7d342-build.20241106-101601.log
+[--:--:--][82d7d342][   build:base.bst                      ] START   Caching artifact
+[00:00:00][82d7d342][   build:base.bst                      ] SUCCESS Caching artifact
+[00:00:00][82d7d342][   build:base.bst                      ] SUCCESS composition/base/82d7d342-build.20241106-101601.log
+[--:--:--][0babd9cc][   build:libhello.bst                  ] START   composition/libhello/0babd9cc-build.20241106-101601.log
+[--:--:--][0babd9cc][   build:libhello.bst                  ] START   Staging dependencies at: /
+[00:00:00][0babd9cc][   build:libhello.bst                  ] SUCCESS Staging dependencies at: /
+[--:--:--][0babd9cc][   build:libhello.bst                  ] START   Integrating sandbox
+[00:00:00][0babd9cc][   build:libhello.bst                  ] SUCCESS Integrating sandbox
+[--:--:--][0babd9cc][   build:libhello.bst                  ] START   Staging sources
+[00:00:00][0babd9cc][   build:libhello.bst                  ] SUCCESS Staging sources
+[--:--:--][0babd9cc][   build:libhello.bst                  ] START   Running commands
+
+    make PREFIX="/usr"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][0babd9cc][   build:libhello.bst                  ] SUCCESS Running commands
+[--:--:--][0babd9cc][   build:libhello.bst                  ] START   Caching artifact
+[00:00:00][0babd9cc][   build:libhello.bst                  ] SUCCESS Caching artifact
+[00:00:00][0babd9cc][   build:libhello.bst                  ] SUCCESS composition/libhello/0babd9cc-build.20241106-101601.log
+[--:--:--][6fb0f24a][   build:hello.bst                     ] START   composition/hello/6fb0f24a-build.20241106-101602.log
+[--:--:--][6fb0f24a][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][6fb0f24a][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][6fb0f24a][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][6fb0f24a][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][6fb0f24a][   build:hello.bst                     ] START   Staging sources
+[00:00:00][6fb0f24a][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][6fb0f24a][   build:hello.bst                     ] START   Running commands
+
+    make PREFIX="/usr"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][6fb0f24a][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][6fb0f24a][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][6fb0f24a][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:00][6fb0f24a][   build:hello.bst                     ] SUCCESS composition/hello/6fb0f24a-build.20241106-101602.log
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] START   composition/runtime-only/c22b89ed-build.20241106-101603.log
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] START   Staging dependencies
+[00:00:00][c22b89ed][   build:runtime-only.bst              ] SUCCESS Staging dependencies
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] START   Computing split
+[00:00:00][c22b89ed][   build:runtime-only.bst              ] SUCCESS Computing split
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] START   Integrating sandbox
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] START   Running commands
+
+    ldconfig "/usr/lib"
+
+[00:00:00][c22b89ed][   build:runtime-only.bst              ] SUCCESS Running commands
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] INFO    Integration added 0 and removed 0 files
+[00:00:00][c22b89ed][   build:runtime-only.bst              ] SUCCESS Integrating sandbox
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] START   Creating composition
+
+    Including files from domains: runtime
+    Excluding orphaned files
+
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] INFO    Composing 287 files
+[00:00:00][c22b89ed][   build:runtime-only.bst              ] SUCCESS Creating composition
+[--:--:--][c22b89ed][   build:runtime-only.bst              ] START   Caching artifact
+[00:00:00][c22b89ed][   build:runtime-only.bst              ] SUCCESS Caching artifact
+[00:00:00][c22b89ed][   build:runtime-only.bst              ] SUCCESS composition/runtime-only/c22b89ed-build.20241106-101603.log
+[00:00:09][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       5
+    Session:     5
+    Fetch Queue: processed 4, skipped 1, failed 0 
+    Build Queue: processed 5, skipped 0, failed 0
+
+

As you can see in the output, this composition has only a few hundred +files, but the complete alpine.bst runtime has several thousand +files.

+
+
+

1.3.2. List the content

+

At the risk of this being a long list, let’s list the +contents of this artifact

+ +
+user@host:~/composition$ bst artifact list-contents runtime-only.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888164.461115    1634 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+  runtime-only.bst:
+	lib
+	lib/ld-musl-x86_64.so.1
+	usr
+	usr/bin
+	usr/bin/2to3
+	usr/bin/2to3-3.6
+	usr/bin/[
+	usr/bin/[[
+	usr/bin/aclocal
+	usr/bin/aclocal-1.15
+	usr/bin/addr2line
+	usr/bin/ar
+	usr/bin/as
+	usr/bin/autoconf
+	usr/bin/autoheader
+	usr/bin/autom4te
+	usr/bin/automake
+	usr/bin/automake-1.15
+	usr/bin/autoreconf
+	usr/bin/autoscan
+	usr/bin/autoupdate
+	usr/bin/awk
+	usr/bin/basename
+	usr/bin/beep
+	usr/bin/blkdiscard
+	usr/bin/bunzip2
+	usr/bin/bzcat
+	usr/bin/bzip2
+	usr/bin/c++filt
+	usr/bin/c_rehash
+	usr/bin/cal
+	usr/bin/cc
+	usr/bin/ccmake
+	usr/bin/chvt
+	usr/bin/cksum
+	usr/bin/clear
+	usr/bin/cmake
+	usr/bin/cmp
+	usr/bin/comm
+	usr/bin/cpack
+	usr/bin/cpio
+	usr/bin/crontab
+	usr/bin/cryptpw
+	usr/bin/ctest
+	usr/bin/cut
+	usr/bin/dc
+	usr/bin/deallocvt
+	usr/bin/diff
+	usr/bin/dirname
+	usr/bin/dos2unix
+	usr/bin/du
+	usr/bin/dumpleases
+	usr/bin/dwp
+	usr/bin/easy_install-3.6
+	usr/bin/eject
+	usr/bin/elfedit
+	usr/bin/env
+	usr/bin/expand
+	usr/bin/expr
+	usr/bin/factor
+	usr/bin/fallocate
+	usr/bin/find
+	usr/bin/flock
+	usr/bin/fold
+	usr/bin/free
+	usr/bin/fuser
+	usr/bin/gdbm_dump
+	usr/bin/gdbm_load
+	usr/bin/gdbmtool
+	usr/bin/getconf
+	usr/bin/getent
+	usr/bin/gprof
+	usr/bin/groups
+	usr/bin/hd
+	usr/bin/head
+	usr/bin/hello
+	usr/bin/hexdump
+	usr/bin/hostid
+	usr/bin/iconv
+	usr/bin/id
+	usr/bin/ifnames
+	usr/bin/install
+	usr/bin/ipcrm
+	usr/bin/ipcs
+	usr/bin/killall
+	usr/bin/ld
+	usr/bin/ld.bfd
+	usr/bin/ldd
+	usr/bin/less
+	usr/bin/logger
+	usr/bin/lsof
+	usr/bin/lsusb
+	usr/bin/lzcat
+	usr/bin/lzma
+	usr/bin/lzopcat
+	usr/bin/m4
+	usr/bin/make
+	usr/bin/md5sum
+	usr/bin/mesg
+	usr/bin/microcom
+	usr/bin/mkfifo
+	usr/bin/mkpasswd
+	usr/bin/nc
+	usr/bin/nl
+	usr/bin/nm
+	usr/bin/nmeter
+	usr/bin/nohup
+	usr/bin/nproc
+	usr/bin/nsenter
+	usr/bin/nslookup
+	usr/bin/objcopy
+	usr/bin/objdump
+	usr/bin/od
+	usr/bin/openvt
+	usr/bin/passwd
+	usr/bin/paste
+	usr/bin/patch
+	usr/bin/perl
+	usr/bin/perl5.26.1
+	usr/bin/perldoc
+	usr/bin/pgrep
+	usr/bin/pip3
+	usr/bin/pip3.6
+	usr/bin/pkill
+	usr/bin/pmap
+	usr/bin/pod2html
+	usr/bin/pod2man
+	usr/bin/pod2text
+	usr/bin/pod2usage
+	usr/bin/podchecker
+	usr/bin/podselect
+	usr/bin/printf
+	usr/bin/pscan
+	usr/bin/pstree
+	usr/bin/pwdx
+	usr/bin/pydoc3
+	usr/bin/pydoc3.6
+	usr/bin/python3
+	usr/bin/python3.6
+	usr/bin/python3.6m
+	usr/bin/pyvenv
+	usr/bin/pyvenv-3.6
+	usr/bin/ranlib
+	usr/bin/readelf
+	usr/bin/readlink
+	usr/bin/realpath
+	usr/bin/renice
+	usr/bin/reset
+	usr/bin/resize
+	usr/bin/scanelf
+	usr/bin/seq
+	usr/bin/setkeycodes
+	usr/bin/setsid
+	usr/bin/sha1sum
+	usr/bin/sha256sum
+	usr/bin/sha3sum
+	usr/bin/sha512sum
+	usr/bin/showkey
+	usr/bin/shred
+	usr/bin/shuf
+	usr/bin/size
+	usr/bin/smemcap
+	usr/bin/sort
+	usr/bin/split
+	usr/bin/ssl_client
+	usr/bin/strings
+	usr/bin/strip
+	usr/bin/sum
+	usr/bin/tac
+	usr/bin/tail
+	usr/bin/tcc
+	usr/bin/tee
+	usr/bin/test
+	usr/bin/time
+	usr/bin/timeout
+	usr/bin/top
+	usr/bin/tr
+	usr/bin/traceroute
+	usr/bin/traceroute6
+	usr/bin/truncate
+	usr/bin/tty
+	usr/bin/ttysize
+	usr/bin/udhcpc6
+	usr/bin/unexpand
+	usr/bin/uniq
+	usr/bin/unix2dos
+	usr/bin/unlink
+	usr/bin/unlzma
+	usr/bin/unlzop
+	usr/bin/unshare
+	usr/bin/unxz
+	usr/bin/unzip
+	usr/bin/uptime
+	usr/bin/uudecode
+	usr/bin/uuencode
+	usr/bin/vi
+	usr/bin/vlock
+	usr/bin/volname
+	usr/bin/wc
+	usr/bin/wget
+	usr/bin/which
+	usr/bin/whoami
+	usr/bin/whois
+	usr/bin/xargs
+	usr/bin/xmlwf
+	usr/bin/xxd
+	usr/bin/xzcat
+	usr/bin/yes
+	usr/lib
+	usr/lib/libarchive.so.13
+	usr/lib/libarchive.so.13.3.2
+	usr/lib/libbfd-2.28.so
+	usr/lib/libbz2.so.1
+	usr/lib/libbz2.so.1.0.6
+	usr/lib/libc.so
+	usr/lib/libcrypto.so.42
+	usr/lib/libcrypto.so.42.0.0
+	usr/lib/libcurl.so.4
+	usr/lib/libcurl.so.4.5.0
+	usr/lib/libexpat.so.1
+	usr/lib/libexpat.so.1.6.7
+	usr/lib/libffi.so.6
+	usr/lib/libffi.so.6.0.4
+	usr/lib/libformw.so.6
+	usr/lib/libformw.so.6.0
+	usr/lib/libgcc_s.so.1
+	usr/lib/libgdbm.so.4
+	usr/lib/libgdbm.so.4.0.0
+	usr/lib/libgdbm_compat.so.4
+	usr/lib/libgdbm_compat.so.4.0.0
+	usr/lib/libhello.so
+	usr/lib/liblz4.so.1
+	usr/lib/liblz4.so.1.8.0
+	usr/lib/liblzma.so.5
+	usr/lib/liblzma.so.5.2.3
+	usr/lib/libmenuw.so.6
+	usr/lib/libmenuw.so.6.0
+	usr/lib/libncursesw.so.6
+	usr/lib/libncursesw.so.6.0
+	usr/lib/libopcodes-2.28.so
+	usr/lib/libpanelw.so.6
+	usr/lib/libpanelw.so.6.0
+	usr/lib/libpython3.6m.so.1.0
+	usr/lib/libpython3.so
+	usr/lib/libreadline.so.7
+	usr/lib/libreadline.so.7.0
+	usr/lib/librhash.so.0
+	usr/lib/libsqlite3.so.0
+	usr/lib/libsqlite3.so.0.8.6
+	usr/lib/libssh2.so.1
+	usr/lib/libssh2.so.1.0.1
+	usr/lib/libssl.so.44
+	usr/lib/libssl.so.44.0.1
+	usr/lib/libstdc++.so.6
+	usr/lib/libstdc++.so.6.0.22
+	usr/lib/libuv.so.1
+	usr/lib/libuv.so.1.0.0
+	usr/sbin
+	usr/sbin/add-shell
+	usr/sbin/addgroup
+	usr/sbin/adduser
+	usr/sbin/arping
+	usr/sbin/brctl
+	usr/sbin/chpasswd
+	usr/sbin/chroot
+	usr/sbin/crond
+	usr/sbin/delgroup
+	usr/sbin/deluser
+	usr/sbin/ether-wake
+	usr/sbin/fbset
+	usr/sbin/fdformat
+	usr/sbin/killall5
+	usr/sbin/loadfont
+	usr/sbin/lspci
+	usr/sbin/nanddump
+	usr/sbin/nandwrite
+	usr/sbin/nbd-client
+	usr/sbin/ntpd
+	usr/sbin/partprobe
+	usr/sbin/powertop
+	usr/sbin/rdate
+	usr/sbin/rdev
+	usr/sbin/readahead
+	usr/sbin/readprofile
+	usr/sbin/remove-shell
+	usr/sbin/rfkill
+	usr/sbin/sendmail
+	usr/sbin/setfont
+	usr/sbin/setlogcons
+	usr/sbin/update-ca-certificates
+
+

Some things to observe here:

+
    +

  • The list does include the /usr/bin/hello program and +also the /usr/lib/libhello.so shared library.

    +

    These paths are both captured by the default split rules +for the runtime domain.

    +
    • +

  • The list does not include the /usr/include/libhello.h +header file which was used to compile /usr/bin/hello.

    +

    The header file is not captured by the runtime domain +by default. It is however captured by the devel domain.

    +
    • +

  • The runtime linker /lib/ld-musl-x86_64.so.1, as this was +explicitly added to the runtime domain for the base/alpine.bst +element which provides this file.

    • +
+
+

Tip

+

The reader at this time might want to list the content of +other elements built from this project, such as the +hello.bst element by itself, or the base/alpine.bst +element.

+
+
+
+

1.3.3. Run the program

+

Finally, lets just run the program we built.

+ +
+user@host:~/composition$ bst shell runtime-only.bst -- hello audience
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888164.894416    1663 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][c22b89ed][    main:runtime-only.bst              ] START   Staging dependencies
+[00:00:00][c22b89ed][    main:runtime-only.bst              ] SUCCESS Staging dependencies
+[--:--:--][c22b89ed][    main:runtime-only.bst              ] START   Integrating sandbox
+[00:00:00][c22b89ed][    main:runtime-only.bst              ] SUCCESS Integrating sandbox
+[--:--:--][c22b89ed][    main:runtime-only.bst              ] STATUS  Running command
+
+    hello audience
+
+Hello audience
+
+

Here we can see that we at least have the required files to run +our hello world program, however we would not have if we were +missing the runtime linker which we added in base/alpine.bst.

+
+
+
+

1.4. Summary

+

In this chapter we’ve gotten familiar with split rules +annotations, and we’ve learned enough about the compose +element such that we can start creating our own compositions using +split domains.

+

We’ve also used the list append directive +and we are now observing the contents of artifacts using +bst artifact list-contents.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/handling-files/filtering.html b/handling-files/filtering.html new file mode 100644 index 000000000..ef2182d8f --- /dev/null +++ b/handling-files/filtering.html @@ -0,0 +1,534 @@ + + + + + + + + + 2. Filtering — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

2. Filtering

+

In this chapter we will explore how to filter the files in an artifact +using the filter element, such that an element might +depend on a subset of the files provided by a filtered element.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/filtering +subdirectory.

+
+
+

2.1. Overview

+

In some cases, it can be useful to depend on a subset of the files of an +element, without depending on the entire element.

+

One scenario where filtering can be useful, is when you have an element which +will build differently depending on what is present in the system where it is +building. In an edge case where a module fails to offer configure time options to +disable an unwanted feature or behavior in the build, you might use +filter elements to ensure that special header files or +pkg-config files are filtered out from the system at build time, such that +the unwanted behavior cannot be built.

+

In many ways, a filter element is like a +compose element, except that it operates on a single +build dependency, without compositing the filtered +element with its runtime dependencies.

+
+

Tip

+

The filter element is special in the sense +that it acts as a window into it’s primary +build dependency.

+

As such, opening a workspace on a +filter element will result in opening a +workspace on the element which it filters. Any other workspace +commands will also be forwarded directly to the filtered element.

+
+
+
+

2.2. Project structure

+

This example again expands on the example presenting in the chapter about +integration commands. In this case +we will modify libhello.bst such that it produces a new file which, +if present, will affect the behavior of it’s reverse dependency hello.bst.

+

Let’s first take a look at how the sources have changed.

+
+

2.2.1. files/hello/Makefile

+
# Sample makefile for hello.c
+#
+.PHONY: all install
+
+all: hello
+
+install:
+	install -d ${DESTDIR}${PREFIX}/bin
+	install -m 755 hello ${DESTDIR}${PREFIX}/bin
+
+hello: hello.c
+	extra_flags="";                                               \
+	if [ -f "${PREFIX}/share/libhello/default-person.txt" ]; then \
+	  extra_flags=-DDEFAULT_PERSON="\"$$(cat ${PREFIX}/share/libhello/default-person.txt)\""; \
+	fi;                                                           \
+	$(CC) $< -o $@ $${extra_flags} -Wall -lhello
+
+
+

Now we have our Makefile discovering the system defined default +person to say hello to.

+
+
+

2.2.2. files/hello/hello.c

+
/*
+ * hello.c - Simple hello program
+ */
+#include <stdio.h>
+#include <libhello.h>
+
+int main(int argc, char *argv[])
+{
+  const char *person = NULL;
+
+  if (argc > 1)
+    person = argv[1];
+
+  if (person)
+    hello(person);
+  else {
+#ifdef DEFAULT_PERSON
+    hello(DEFAULT_PERSON);
+#else
+    hello("stranger");
+#endif
+  }
+
+  return 0;
+}
+
+
+

If this program has been given a DEFAULT_PERSON, then it will +say hello to that person in the absence of any argument.

+
+
+

2.2.3. project.conf

+
# Unique project name
+name: filtering
+
+# Minimum required BuildStream version
+min-version: 2.0
+
+# Subdirectory where elements are stored
+element-path: elements
+
+# Define an alias for our alpine tarball
+aliases:
+  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
+
+# Use an option to decide if we should use the filter
+#
+options:
+  use_filter:
+    type: bool
+    description: Whether to use a filter on the libhello.bst element
+    default: False
+
+
+

Here, we’ve added a project option to decide +whether to use the filter element or not.

+

This is merely for brevity, so that we can demonstrate the behavior +of depending on the filtered element without defining two separate versions +of the hello.bst element.

+
+
+

2.2.4. elements/libhello.bst

+
kind: manual
+description: |
+
+  The libhello library
+
+# Depend on the base system
+depends:
+- base.bst
+
+# Stage the files/libhello directory for building
+sources:
+  - kind: local
+    path: files/libhello
+
+# Now configure the commands to run
+config:
+
+  build-commands:
+  - make PREFIX="%{prefix}"
+
+  install-commands:
+  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
+
+public:
+  bst:
+    # Define a split domain which captures the defaults
+    # which this library installs into %{datadir}
+    #
+    split-rules:
+      defaults:
+      - "%{datadir}/libhello/default-person.txt"
+
+
+

We’ve added some split rules here to declare +a new split domain named defaults, and we’ve added the new +default-person.txt file to this domain.

+
+
+

2.2.5. elements/libhello-filtered.bst

+
kind: filter
+description: |
+
+  A filtered version of libhello which excludes the defaults
+
+# Specify the build dependency to filter
+build-depends:
+- libhello.bst
+
+# Propagate runtime dependencies
+runtime-depends:
+- base.bst
+
+# Now configure the commands to run
+config:
+  exclude:
+  - defaults
+
+
+

And we’ve added a new filter element to the project +which uses the exclude option of the filter configuration.

+

This is essentially a statement that any files mentioned in the the +defaults domain of the libhello.bst element should be excluded from +the resulting artifact.

+
+

Important

+

Notice that you need to explicitly declare any +runtime dependencies which are required by the +resulting artifact of a filter element, as runtime +dependencies of the build dependency are not transient.

+
+
+
+

2.2.6. elements/hello.bst

+
kind: manual
+description: |
+
+  The hello application
+
+# Depend on the hello library, or the filtered version
+#
+(?):
+- use_filter == True:
+    depends:
+      - libhello-filtered.bst
+- use_filter == False:
+    depends:
+      - libhello.bst
+
+# Stage the files/hello directory for building
+sources:
+  - kind: local
+    path: files/hello
+
+# Now configure the commands to run
+config:
+
+  build-commands:
+  - make PREFIX="%{prefix}"
+
+  install-commands:
+  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
+
+
+

Here we’ve merely added a conditional statement +which allows us to test the hello.bst element depending on the filtered +version of the library, or the unfiltered version.

+
+
+
+

2.3. Using the project

+

Let’s just skip over building the hello.bst element with the +use_filter option both True and False, these elements +are easily built with bst build as such:

+
bst --option use_filter True build hello.bst
+bst --option use_filter False build hello.bst
+
+
+
+

2.3.1. Observing the artifacts

+

Let’s take a look at the built artifacts.

+
+

2.3.1.1. libhello.bst

+ +
+user@host:~/filtering$ bst artifact list-contents libhello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888199.731437    2644 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+  libhello.bst:
+	usr
+	usr/include
+	usr/include/libhello.h
+	usr/lib
+	usr/lib/libhello.so
+	usr/share
+	usr/share/libhello
+	usr/share/libhello/default-person.txt
+
+

Here we can see the full content of the libhello.bst artifact.

+
+
+

2.3.1.2. libhello-filtered.bst

+ +
+user@host:~/filtering$ bst artifact list-contents libhello-filtered.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888200.141202    2672 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+  libhello-filtered.bst:
+	usr
+	usr/include
+	usr/include/libhello.h
+	usr/lib
+	usr/lib/libhello.so
+
+

Here we can see that the default-person.txt file has been filtered +out of the libhello.bst artifact when creating the libhello-filtered.bst +artifact.

+
+
+
+

2.3.2. Running hello.bst

+

Now if we run the program built by hello.bst in either build +modes, we can observe the expected behavior.

+
+

2.3.2.1. Run hello.bst built directly against libhello.bst

+ +
+user@host:~/filtering$ bst --option use_filter False shell hello.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888200.558882    2700 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][911b8a6d][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][911b8a6d][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][911b8a6d][    main:hello.bst                     ] START   Integrating sandbox
+[--:--:--][        ][    main:hello.bst                     ] START   Running commands
+
+    ldconfig "/usr/lib"
+
++ sh -e -c ldconfig "/usr/lib"
+[00:00:00][        ][    main:hello.bst                     ] SUCCESS Running commands
+[00:00:00][911b8a6d][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][911b8a6d][    main:hello.bst                     ] STATUS  Running command
+
+    hello
+
+Hello Sophia
+
+

Here we can see that the hello world program is using the system +configured default person to say hello to.

+
+
+

2.3.2.2. Run hello.bst built against libhello-filtered.bst

+ +
+user@host:~/filtering$ bst --option use_filter True shell hello.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888201.788871    2765 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][db2ce036][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][db2ce036][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][db2ce036][    main:hello.bst                     ] START   Integrating sandbox
+[--:--:--][        ][    main:hello.bst                     ] START   Running commands
+
+    ldconfig "/usr/lib"
+
++ sh -e -c ldconfig "/usr/lib"
+[00:00:00][        ][    main:hello.bst                     ] SUCCESS Running commands
+[00:00:00][db2ce036][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][db2ce036][    main:hello.bst                     ] STATUS  Running command
+
+    hello
+
+Hello stranger
+
+

And now we’re reverting to the behavior we have when no +system configured default person was installed at build time.

+
+
+
+
+

2.4. Summary

+

In this chapter, we’ve introduced the filter +element which allows one to filter the output of an element and +effectively create a dependency on a subset of an element’s files.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/handling-files/overlaps.html b/handling-files/overlaps.html new file mode 100644 index 000000000..de443428f --- /dev/null +++ b/handling-files/overlaps.html @@ -0,0 +1,374 @@ + + + + + + + + + 3. Overlapping files — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

3. Overlapping files

+

In this chapter, we will discuss what happens when files from multiple +element artifacts conflict with eachother, and what strategies we can +use to resolve these situations.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/overlaps +subdirectory.

+
+
+

3.1. Overview

+

This project builds on the previous chapter on composition, +and as such we’ll only go over what has changed from there, which is not much.

+
+
+

3.2. Project structure

+

In this example we’ve just extended the libhello.bst and the hello.bst +elements such that they both install an additional file: %{docdir}/hello.txt.

+

We’ve updated the following Makefiles:

+
+

3.2.1. files/libhello/Makefile

+
# Sample makefile for hello library
+#
+.PHONY: all install
+
+all: libhello.so
+
+install:
+	install -d ${DESTDIR}${PREFIX}/lib
+	install -d ${DESTDIR}${PREFIX}/include
+	install -d ${DESTDIR}${PREFIX}/share/doc
+	install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
+	install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
+	install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc
+
+%.o: %.c %.h
+	$(CC) -c $< -o $@ -Wall
+
+libhello.so: libhello.o
+	$(CC) -shared -o $@ $<
+
+
+
+
+

3.2.2. files/hello/Makefile

+
# Sample makefile for hello.c
+#
+.PHONY: all install
+
+all: hello
+
+install:
+	install -d ${DESTDIR}${PREFIX}/bin
+	install -d ${DESTDIR}${PREFIX}/share/doc
+	install -m 755 hello ${DESTDIR}${PREFIX}/bin
+	install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc
+
+hello: hello.c
+	$(CC) $< -o $@ -Wall -lhello
+
+
+

As you can see, this now presents a conflict of sorts, where multiple +elements in the pipeline want to install the same file.

+
+
+
+

3.3. Using the project

+

In this chapter, we’re only going to present the warning and then +discuss how to mitigate this situation.

+

See what happens when we try to build the runtime-only.bst +compose element:

+ +
+user@host:~/overlaps$ bst build runtime-only.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888259.185127    6749 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:17:39
+    Project:       overlaps (/home/user/buildstream/doc/examples/overlaps)
+    Targets:       runtime-only.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-b4nf6j9j/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-b4nf6j9j
+    Log Files:               /home/user/buildstream/doc/run-bst-b4nf6j9j/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-b4nf6j9j/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-b4nf6j9j/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: overlaps
+
+    Element Plugins
+        compose: core plugin
+        manual:  core plugin
+        stack:   core plugin
+        import:  core plugin
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Pipeline
+      cached 617c4e863e3404763b4b4860156e4eb510ea5de1f99384d71465f79ee9495504 base/alpine.bst 
+      cached 1f56d7f344afc5f3f3738600e43bd523ed68d3208419e3d49d1325ff0b83a843 base.bst 
+      cached eb9542a1a6a86e71e839554cf67aca2e2fa52c29eefbb8707889bae8ebd9d429 libhello.bst 
+      cached 5ba078b2324d29d7a455f3a1e754443279bd62fc685c0849bb86ef214cbc51aa hello.bst 
+   buildable 06c0be7b5c5692db9a37354e4e1210104d4ebd717f35d49e03ff7508ba3e1bb2 runtime-only.bst 
+===============================================================================
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   overlaps/runtime-only/06c0be7b-build.20241106-101739.log
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Staging dependencies
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] WARNING [overlaps]: Non-whitelisted overlaps detected
+
+    Staged files overwrite existing files in staging area: /
+    /usr/share/doc/hello.txt: hello.bst is not permitted to overlap other elements, order hello.bst above libhello.bst 
+
+[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Staging dependencies
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Computing split
+[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Computing split
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Integrating sandbox
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Running commands
+
+    ldconfig "/usr/lib"
+
+[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Running commands
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] INFO    Integration added 0 and removed 0 files
+[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Integrating sandbox
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Creating composition
+
+    Including files from domains: runtime
+    Excluding orphaned files
+
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] INFO    Composing 287 files
+[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Creating composition
+[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Caching artifact
+[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Caching artifact
+[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS overlaps/runtime-only/06c0be7b-build.20241106-101739.log
+[00:00:01][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       5
+    Session:     5
+    Fetch Queue: processed 0, skipped 5, failed 0 
+    Build Queue: processed 1, skipped 4, failed 0
+
+

Notice the warning message about the conflicting file, it is there to +inform the user about which files are overlapping, and also which +elements are being staged in which order.

+

Note also that BuildStream does not discover the overlap until the +moment that you build a reverse dependency which will require staging +of both artifacts.

+
+

Tip

+

The overlaps warning discussed here can be configured to be +a fatal warning. This is useful +in the case that you want to be strict about avoiding overlapping +files in your project.

+
+
+
+

3.4. Mitigating overlapping files

+

Since we recently discussed filtering of artifacts, +we should note that it is of course possible to handle this case by +having hello.bst depend on a filtered version of libhello.bst with the +offending file excluded.

+

However, working with filter elements just for the +sake of handling a conflicting artifact would be quite inconvenient, so we +have other means.

+
+

3.4.1. Whitelisting overlapping files

+

BuildStream allows explicitly ignoring such errors by adding the files +to an overlap whitelist, you could achieve +this in the given example by adding the following to the hello.bst element:

+
public:
+  bst:
+    overlap-whitelist:
+    - |
+      %{docdir}/hello.txt
+
+
+
+

Note

+

Note that glob patterns are also +supported in the whitelist.

+
+
+
+

3.4.2. Artifact munging

+

Another way around this situation is the munge the artifacts at install +time such that there is no conflict.

+

This is the easiest approach in the case that you might want to keep the +underlying %{docdir}/hello.txt from libhello.bst and discard the +same file from hello.bst.

+

In this case, we might modify the hello.bst file so that it’s install +command contain an rm statement, as such:

+
install-commands:
+- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
+
+- |
+  # Rid ourselves of the unwanted file at install time
+  rm -f %{install-root}%{docdir}/hello.txt
+
+
+

This would cause later builds of runtime-only.bst to no longer +conflict on the given file.

+
+
+
+

3.5. Summary

+

In this chapter we’ve presented a situation where an artifact +can conflict with another artifact by way of providing the +same files.

+

We’ve presented the overlap whitelist +public data which is the typical solution for silencing the +error when the outcome is desired, and also presented a strategy +to deal with cases where you want to keep files from the +overlapped artifact instead.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/index.html b/index.html new file mode 100644 index 000000000..4de6d620f --- /dev/null +++ b/index.html @@ -0,0 +1,132 @@ + + + + + + + + + BuildStream documentation — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

BuildStream documentation

+

These docs cover everything you need to build and integrate software stacks +using BuildStream.

+

They begin with a basic introduction to BuildStream, background +information on basic concepts, and a guide to the BuildStream command line interface. +Later sections provide detailed information on BuildStream internals.

+
+ +
+

For any other information, including how to install BuildStream, +refer to the BuildStream website.

+
+ + +
+
+
+ +
+ +
+

© Copyright 2017-2022, The Apache Software Foundation.

+
+ + Built with Sphinx using a + theme + provided by Read the Docs. + + +
+
+
+
+
+ + + + \ No newline at end of file diff --git a/junctions/junction-elements.html b/junctions/junction-elements.html new file mode 100644 index 000000000..fcf23408f --- /dev/null +++ b/junctions/junction-elements.html @@ -0,0 +1,380 @@ + + + + + + + + + 1. Junction elements — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

1. Junction elements

+

BuildStream’s junction elements are the mechanism which +allow projects to interact and depend on eachother.

+

Junction elements represent the BuildStream project you are depending, and behave +much like other elements in the sense that they can be fetched +and tracked like other elements, except that regular +elements cannot depend on junctions directly, nor can junctions be built. +Instead, junctions act like a window into another project you depend on, and allow +elements of your project to depend on elements exposed by the project referenced by +the junction.

+

Projects which are junctioned by your project are referred to as subprojects.

+
+

1.1. A simple example

+
+

Note

+

This example is distributed with BuildStream in the +doc/examples/junctions +subdirectory.

+
+

Below is a simple example of bst file for the junction element, which +we have called hello-junction.bst in this project:

+
kind: junction
+sources:
+- kind: local
+  path: autotools
+
+
+

This element imports the autotools example subproject distributed with BuildStream +in the doc/examples/junctions/autotools +subdirectory.

+
+

Note

+

For the sake of this example we are using a local source in a subdirectory +of the example project.

+

Since junctions allow interoperability of projects, it would be more common +to use a junction to a remote project under separate revision control, possibly +using a kind: git source.

+
+

The below bst file describes the element callHello.bst, which depends on the +hello.bst element from the autotools example:

+
kind: import
+
+sources:
+- kind: local
+  path: files/callHello.sh
+
+depends:
+  - filename: hello.bst
+    junction: hello-junction.bst
+
+
+

Note how this element refers to the previously declared hello-junction.bst +junction in its dependency dictionary. This dependency +expresses that we are depending on the hello.bst element in the project +which hello-junction.bst refers to.

+

The callHello.bst element simply imports a callHello.sh shell script which +calls the hello command provided by hello.bst:

+
#!/bin/sh
+echo "Calling hello:"
+hello
+
+
+
+
+

1.2. Building and running

+

Building the callHello.bst element which requires an external project +is just a matter of invoking bst build in the +regular way:

+ +
+user@host:~/junctions$ bst build callHello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888235.804244    5363 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[--:--:--][6d2e5da0][   fetch:hello-junction.bst            ] START   junctions/hello-junction/6d2e5da0-fetch.20241106-101715.log
+[00:00:00][6d2e5da0][   fetch:hello-junction.bst            ] SUCCESS junctions/hello-junction/6d2e5da0-fetch.20241106-101715.log
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:17:15
+    Project:       junctions (/home/user/buildstream/doc/examples/junctions)
+    Targets:       callHello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-phxqlxaq/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-phxqlxaq
+    Log Files:               /home/user/buildstream/doc/run-bst-phxqlxaq/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-phxqlxaq/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-phxqlxaq/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: junctions
+
+    Element Plugins
+        junction: core plugin
+        import:   core plugin
+
+    Source Plugins
+        local: core plugin
+
+Project: autotools
+    Junction path: hello-junction.bst
+    Loaded by:     callHello.bst [line 8 column 4]
+
+    Element Plugins
+        autotools: python package 'buildstream-plugins 2.2.0' at: /home/user/buildstream/.tox/docs/lib/python3.12/site-packages
+        stack:     core plugin
+        import:    core plugin
+
+    Source Plugins
+        tar: core plugin
+
+Pipeline
+fetch needed 6a78c03097648e558c803c20d8d20f61d3e87e3e616f3a45923704cfe95019af hello-junction.bst:base/alpine.bst 
+     waiting 644e964abe673c8a8723cf5a3fceeeda44a7bea364f55604fc8132af44a4fb59 hello-junction.bst:base.bst 
+fetch needed ffeab61203b5940714c33b1cc29822b9391b9fea5876faef23c024bf6acd99cf hello-junction.bst:hello.bst 
+     waiting b6da6abe68420499ec616c50433619027a915e37217b85450a5590a28bd28820 callHello.bst 
+===============================================================================
+[--:--:--][644e964a][   fetch:hello-junction.bst:base.bst   ] START   autotools/base/644e964a-fetch.20241106-101715.log
+[--:--:--][6a78c030][   fetch:hello-junction.bst:base/alpine.bst] START   autotools/base-alpine/6a78c030-fetch.20241106-101715.log
+[--:--:--][ffeab612][   fetch:hello-junction.bst:hello.bst  ] START   autotools/hello/ffeab612-fetch.20241106-101715.log
+[--:--:--][b6da6abe][   fetch:callHello.bst                 ] START   junctions/callHello/b6da6abe-fetch.20241106-101715.log
+[--:--:--][ffeab612][   fetch:hello-junction.bst:hello.bst  ] START   Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
+[--:--:--][6a78c030][   fetch:hello-junction.bst:base/alpine.bst] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:00][644e964a][   fetch:hello-junction.bst:base.bst   ] SUCCESS autotools/base/644e964a-fetch.20241106-101715.log
+[00:00:00][b6da6abe][   fetch:callHello.bst                 ] SUCCESS junctions/callHello/b6da6abe-fetch.20241106-101715.log
+[00:00:00][6a78c030][   fetch:hello-junction.bst:base/alpine.bst] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:00][ffeab612][   fetch:hello-junction.bst:hello.bst  ] SUCCESS Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
+[00:00:01][ffeab612][   fetch:hello-junction.bst:hello.bst  ] SUCCESS autotools/hello/ffeab612-fetch.20241106-101715.log
+[00:00:06][6a78c030][   fetch:hello-junction.bst:base/alpine.bst] SUCCESS autotools/base-alpine/6a78c030-fetch.20241106-101715.log
+[--:--:--][6a78c030][   build:hello-junction.bst:base/alpine.bst] START   autotools/base-alpine/6a78c030-build.20241106-101722.log
+[--:--:--][6a78c030][   build:hello-junction.bst:base/alpine.bst] START   Staging sources
+[00:00:00][6a78c030][   build:hello-junction.bst:base/alpine.bst] SUCCESS Staging sources
+[--:--:--][6a78c030][   build:hello-junction.bst:base/alpine.bst] START   Caching artifact
+[00:00:00][6a78c030][   build:hello-junction.bst:base/alpine.bst] SUCCESS Caching artifact
+[00:00:00][6a78c030][   build:hello-junction.bst:base/alpine.bst] SUCCESS autotools/base-alpine/6a78c030-build.20241106-101722.log
+[--:--:--][644e964a][   build:hello-junction.bst:base.bst   ] START   autotools/base/644e964a-build.20241106-101722.log
+[--:--:--][644e964a][   build:hello-junction.bst:base.bst   ] START   Caching artifact
+[00:00:00][644e964a][   build:hello-junction.bst:base.bst   ] SUCCESS Caching artifact
+[00:00:00][644e964a][   build:hello-junction.bst:base.bst   ] SUCCESS autotools/base/644e964a-build.20241106-101722.log
+[--:--:--][ffeab612][   build:hello-junction.bst:hello.bst  ] START   autotools/hello/ffeab612-build.20241106-101722.log
+[--:--:--][ffeab612][   build:hello-junction.bst:hello.bst  ] START   Staging dependencies at: /
+[00:00:00][ffeab612][   build:hello-junction.bst:hello.bst  ] SUCCESS Staging dependencies at: /
+[--:--:--][ffeab612][   build:hello-junction.bst:hello.bst  ] START   Integrating sandbox
+[00:00:00][ffeab612][   build:hello-junction.bst:hello.bst  ] SUCCESS Integrating sandbox
+[--:--:--][ffeab612][   build:hello-junction.bst:hello.bst  ] START   Staging sources
+[00:00:00][ffeab612][   build:hello-junction.bst:hello.bst  ] SUCCESS Staging sources
+[--:--:--][ffeab612][   build:hello-junction.bst:hello.bst  ] START   Running commands
+
+    export NOCONFIGURE=1;
+    
+    if [ -x ./configure ]; then true;
+    elif [ -x ./autogen ]; then ./autogen;
+    elif [ -x ./autogen.sh ]; then ./autogen.sh;
+    elif [ -x ./bootstrap ]; then ./bootstrap;
+    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
+    else autoreconf -ivf .;
+    fi
+    ./configure --prefix=/usr \
+    --exec-prefix=/usr \
+    --bindir=/usr/bin \
+    --sbindir=/usr/sbin \
+    --sysconfdir=/etc \
+    --datadir=/usr/share \
+    --includedir=/usr/include \
+    --libdir=/usr/lib \
+    --libexecdir=/usr/libexec \
+    --localstatedir=/var \
+    --sharedstatedir=/usr/com \
+    Message contains 23 additional lines
+
+[00:00:03][ffeab612][   build:hello-junction.bst:hello.bst  ] SUCCESS Running commands
+[--:--:--][ffeab612][   build:hello-junction.bst:hello.bst  ] START   Caching artifact
+[00:00:00][ffeab612][   build:hello-junction.bst:hello.bst  ] SUCCESS Caching artifact
+[00:00:04][ffeab612][   build:hello-junction.bst:hello.bst  ] SUCCESS autotools/hello/ffeab612-build.20241106-101722.log
+[--:--:--][b6da6abe][   build:callHello.bst                 ] START   junctions/callHello/b6da6abe-build.20241106-101726.log
+[--:--:--][b6da6abe][   build:callHello.bst                 ] START   Staging sources
+[00:00:00][b6da6abe][   build:callHello.bst                 ] SUCCESS Staging sources
+[--:--:--][b6da6abe][   build:callHello.bst                 ] START   Caching artifact
+[00:00:00][b6da6abe][   build:callHello.bst                 ] SUCCESS Caching artifact
+[00:00:00][b6da6abe][   build:callHello.bst                 ] SUCCESS junctions/callHello/b6da6abe-build.20241106-101726.log
+[00:00:10][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       4
+    Session:     4
+    Fetch Queue: processed 4, skipped 0, failed 0 
+    Build Queue: processed 4, skipped 0, failed 0
+
+

You can see that the hello.bst element and its dependencies from the autotools +project have been built as a part of the pipeline for callHello.bst.

+

We can now invoke bst shell and run our callHello.sh +script, which in turn also calls the hello program installed by the +subproject’s hello.bst element.

+ +
+user@host:~/junctions$ bst shell callHello.bst -- /bin/sh callHello.sh
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888247.185803    6452 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][b6da6abe][    main:callHello.bst                 ] START   Staging dependencies
+[00:00:00][b6da6abe][    main:callHello.bst                 ] SUCCESS Staging dependencies
+[--:--:--][b6da6abe][    main:callHello.bst                 ] START   Integrating sandbox
+[00:00:00][b6da6abe][    main:callHello.bst                 ] SUCCESS Integrating sandbox
+[--:--:--][b6da6abe][    main:callHello.bst                 ] STATUS  Running command
+
+    /bin/sh callHello.sh
+
+Calling hello:
+Hello World!
+This is amhello 1.0.
+
+
+
+

1.3. Further reading

+

For an example of junction elements being used in a real project, take a look +at the freedesktop-sdk junction +in the gnome-build-meta project.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/junctions/junction-includes.html b/junctions/junction-includes.html new file mode 100644 index 000000000..2a516b506 --- /dev/null +++ b/junctions/junction-includes.html @@ -0,0 +1,696 @@ + + + + + + + + + 3. Subproject includes — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

3. Subproject includes

+

We’ve already discussed how we can add optionality to projects and +explored how we can perform conditional statements and include fragments +of BuildStream YAML in the earlier chapter about optionality and +directives.

+

In this chapter we’re going to explore how we can use include directives to include YAML fragments from a subproject +referred to by a junction element, and how +project options can be specified in the configuration +of your junction.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/junction-includes +subdirectory.

+
+
+

3.1. Overview

+

It is a goal of BuildStream to provide developers and integrators with the tools +they need to maintain software stacks which depend on eachother with least friction +as possible, such that one can integrate upgrades of projects one depends on +via junction elements regularly and with the least +hassle as possible.

+

Project options and include directives combined form the basis on which projects can +maximize on code sharing effectively, and the basis on which BuildStream +projects can form reliable APIs.

+
+

3.1.1. Project options

+

The options which a project exposes is a fairly +limited API surface, it allows one to configure a limited set of options +advertized by the project maintainers, and the options will affect what +kind of artifacts will be produced by the project.

+

This kind of optionality however does not allow consumers to entirely +redefine how artifacts are produced and how elements are configured.

+

On the one hand, this limitation can be frustrating, as one constantly +finds themselves requiring a feature that their subproject does not +support right now. On the other hand, the limitation of features which +a given project chooses to support is what guards downstream project +consumers against consuming artifacts which are not supported by the upstream.

+

Project options are designed to enforce a separation of concerns, +where we expect that downstreams will either fork a project in order +to support a new feature, or convince the upstream to start supporting +a new feature. Furthermore, limited API surfaces for interdependent +projects offers a possibility of API stability of projects, such +that you can upgrade your dependencies with limited friction.

+
+
+

3.1.2. Includes

+

The includes which a project might advertize +as “public”, form the output of the API exchange between a project and +its subproject(s).

+

Cross-project include files allow a project to inherit configuration from +a subproject. Include files can be used to define anything from the +variables one needs to have in context in order to +build into or link into alternative system prefixes, what special compiler flags +to use when building for a specific machine architecture, to customized +shell configurations to use when testing out applications +in bst shell.

+

This chapter will provide an example of the mechanics of cross project +includes when combined with project optionality.

+
+
+
+

3.2. Project structure

+
+

3.2.1. Project options

+

This example is comprised of two separate projects, both of which offer +some project options. This is intended to emphasize how your toplevel project +options can be used to select and configure options to use in the subprojects +you depend on.

+

For convenience, the subproject is stored in the subdirectory of +the toplevel project, while in the real world the subproject is probably +hosted elsewhere.

+

First let’s take a look at the options declared in the respective +project.conf files.

+
+

3.2.1.1. Toplevel project.conf

+
name: project
+min-version: 2.0
+element-path: elements
+
+aliases:
+  gnu: http://ftpmirror.gnu.org/gnu/automake/
+
+plugins:
+- origin: pip
+  package-name: buildstream-plugins
+  elements:
+  - autotools
+
+# Define some options for this project
+#
+options:
+  funky:
+    type: bool
+    description: Whether this project is funky
+    default: False
+
+
+
+
+

3.2.1.2. Subproject project.conf

+
name: subproject
+min-version: 2.0
+element-path: elements
+
+aliases:
+  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
+
+# Define some options for this project
+#
+options:
+  color:
+    type: enum
+    description: The color of this runtime
+    values:
+    - red
+    - green
+    - blue
+    default: blue
+
+
+

As we can see, these two projects both offer some arbitrarily named options.

+
+
+
+

3.2.2. Conditional configuration of subproject

+

The toplevel project here does some conditional configuration of the +subproject.

+
+

3.2.2.1. Toplevel elements/subproject-junction.bst

+
kind: junction
+
+config:
+  # Configure the options for subproject
+  #
+  # If our project is funky, then it requires
+  # a blue subproject, otherwise we use a red one.
+  #
+  options:
+    color: red
+    (?):
+    - funky == True:
+        color: blue
+
+sources:
+- kind: local
+  path: subproject
+
+
+

Here we can see that projects can use +conditional statements to make +decisions about subproject configuration based on their own configuration.

+

In this example, if the toplevel project is funky, then it will +configure its subproject with color set to blue, otherwise it +will use the red variant of the subproject color.

+
+
+
+

3.2.3. Including configuration from a subproject

+

Here there are a couple of aspects to observe, namely how the +toplevel project includes files across a junction boundary, +and how that include file might be implemented.

+
+

3.2.3.1. Toplevel elements/hello.bst

+
kind: autotools
+description: |
+
+    Hello world example from automake
+
+variables:
+
+  # The special paths.bst from our subproject is used to
+  # define the paths of some elements in this project.
+  #
+  (@): subproject-junction.bst:include/paths.bst
+
+  # The hello world example lives in the doc/amhello folder.
+  command-subdir: doc/amhello
+
+sources:
+- kind: tar
+  url: gnu:automake-1.16.tar.gz
+  ref: 80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f
+
+depends:
+  - filename: base.bst
+    junction: subproject-junction.bst
+
+
+

Here we can see the same element which we discussed in the +autotools example, except that we’re including +a file from the subproject. As explained in the reference manual, +this is done by prefixing the include path with the local junction +element name and then a colon.

+

Note that in this case, the API contract is simply that hello.bst is +including paths.bst, and has the expectation that paths.bst will +in some way influence the variables, nothing more.

+

It can be that an include file is expected to create new variables, and +it can be that the subproject might declare things differently depending +on the subproject’s own configuration, as we will observe next.

+
+
+

3.2.3.2. Subproject include/paths.bst

+
# When this project color is blue, including this
+# file causes installations to be relocated to /opt
+#
+prefix: /usr
+(?):
+- color == "blue":
+    prefix: /opt
+
+
+

Here, we can see the include file itself is making a +conditional statement, in turn +deciding what values to use depending on how the project was configured.

+

This decision will provide valuable context for any file including paths.bst, +whether it be an element, a project.conf which applies the variable as +a default for the entire project, whether it is being included by files +in the local project, or whether it is being included by a downstream +project which junctions this project, as is the case in this example.

+
+
+
+
+

3.3. Using the project

+

At this stage, you have probably already reasoned out what would happen +if we tried to build and run the project.

+

Nevertheless, we will still present the outputs here for observation.

+
+

3.3.1. Building the project normally

+

Here we build the project without any special arguments.

+ +
+user@host:~/junction-includes$ bst build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888218.337185    3181 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:16:58
+    Project:       project (/home/user/buildstream/doc/examples/junction-includes)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-f15gxvq4/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-f15gxvq4
+    Log Files:               /home/user/buildstream/doc/run-bst-f15gxvq4/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-f15gxvq4/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-f15gxvq4/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: project
+
+    Project Options
+        funky: 0
+
+    Element Plugins
+        junction:  core plugin
+        autotools: python package 'buildstream-plugins 2.2.0' at: /home/user/buildstream/.tox/docs/lib/python3.12/site-packages
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Project: subproject
+    Junction path: subproject-junction.bst
+    Loaded by:     hello.bst [line 11 column 7]
+
+    Project Options
+        color: red
+
+    Element Plugins
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        tar: core plugin
+
+Pipeline
+fetch needed 2130295a58bedd83aaf3998edb121ccd5942cd749f14b87c19a4be330c90d54f subproject-junction.bst:base/alpine.bst 
+     waiting c93bc58b2fe8afba05ea49c1a941775a15442a56ea826dee5c017641de7c469c subproject-junction.bst:base.bst 
+     waiting 75ba510d9aa73503f87fd9bc44db651a2cf7aca8c67f13d686abbe0b35dced0e hello.bst 
+===============================================================================
+[--:--:--][c93bc58b][   fetch:subproject-junction.bst:base.bst] START   subproject/base/c93bc58b-fetch.20241106-101658.log
+[--:--:--][2130295a][   fetch:subproject-junction.bst:base/alpine.bst] START   subproject/base-alpine/2130295a-fetch.20241106-101658.log
+[--:--:--][2130295a][   fetch:subproject-junction.bst:base/alpine.bst] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:00][c93bc58b][   fetch:subproject-junction.bst:base.bst] SUCCESS subproject/base/c93bc58b-fetch.20241106-101658.log
+[00:00:00][2130295a][   fetch:subproject-junction.bst:base/alpine.bst] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:06][2130295a][   fetch:subproject-junction.bst:base/alpine.bst] SUCCESS subproject/base-alpine/2130295a-fetch.20241106-101658.log
+[--:--:--][2130295a][   build:subproject-junction.bst:base/alpine.bst] START   subproject/base-alpine/2130295a-build.20241106-101705.log
+[--:--:--][2130295a][   build:subproject-junction.bst:base/alpine.bst] START   Staging sources
+[00:00:00][2130295a][   build:subproject-junction.bst:base/alpine.bst] SUCCESS Staging sources
+[--:--:--][2130295a][   build:subproject-junction.bst:base/alpine.bst] START   Caching artifact
+[00:00:00][2130295a][   build:subproject-junction.bst:base/alpine.bst] SUCCESS Caching artifact
+[00:00:00][2130295a][   build:subproject-junction.bst:base/alpine.bst] SUCCESS subproject/base-alpine/2130295a-build.20241106-101705.log
+[--:--:--][c93bc58b][   build:subproject-junction.bst:base.bst] START   subproject/base/c93bc58b-build.20241106-101705.log
+[--:--:--][c93bc58b][   build:subproject-junction.bst:base.bst] START   Caching artifact
+[00:00:00][c93bc58b][   build:subproject-junction.bst:base.bst] SUCCESS Caching artifact
+[00:00:00][c93bc58b][   build:subproject-junction.bst:base.bst] SUCCESS subproject/base/c93bc58b-build.20241106-101705.log
+[--:--:--][75ba510d][   build:hello.bst                     ] START   project/hello/75ba510d-build.20241106-101705.log
+[--:--:--][75ba510d][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][75ba510d][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][75ba510d][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][75ba510d][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][75ba510d][   build:hello.bst                     ] START   Staging sources
+[00:00:00][75ba510d][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][75ba510d][   build:hello.bst                     ] START   Running commands
+
+    export NOCONFIGURE=1;
+    
+    if [ -x ./configure ]; then true;
+    elif [ -x ./autogen ]; then ./autogen;
+    elif [ -x ./autogen.sh ]; then ./autogen.sh;
+    elif [ -x ./bootstrap ]; then ./bootstrap;
+    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
+    else autoreconf -ivf .;
+    fi
+    ./configure --prefix=/usr \
+    --exec-prefix=/usr \
+    --bindir=/usr/bin \
+    --sbindir=/usr/sbin \
+    --sysconfdir=/etc \
+    --datadir=/usr/share \
+    --includedir=/usr/include \
+    --libdir=/usr/lib \
+    --libexecdir=/usr/libexec \
+    --localstatedir=/var \
+    --sharedstatedir=/usr/com \
+    Message contains 23 additional lines
+
+[00:00:03][75ba510d][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][75ba510d][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][75ba510d][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:03][75ba510d][   build:hello.bst                     ] SUCCESS project/hello/75ba510d-build.20241106-101705.log
+[00:00:10][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 2, skipped 1, failed 0 
+    Build Queue: processed 3, skipped 0, failed 0
+
+
+
+

3.3.2. Building the project in funky mode

+

Now let’s see what happens when we build the project in funky mode

+ +
+user@host:~/junction-includes$ bst --option funky True build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888229.623083    4195 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:17:09
+    Project:       project (/home/user/buildstream/doc/examples/junction-includes)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-f15gxvq4/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-f15gxvq4
+    Log Files:               /home/user/buildstream/doc/run-bst-f15gxvq4/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-f15gxvq4/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-f15gxvq4/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: project
+
+    Project Options
+        funky: 1
+
+    Element Plugins
+        junction:  core plugin
+        autotools: python package 'buildstream-plugins 2.2.0' at: /home/user/buildstream/.tox/docs/lib/python3.12/site-packages
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Project: subproject
+    Junction path: subproject-junction.bst
+    Loaded by:     hello.bst [line 11 column 7]
+
+    Project Options
+        color: blue
+
+    Element Plugins
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        tar: core plugin
+
+Pipeline
+      cached 2130295a58bedd83aaf3998edb121ccd5942cd749f14b87c19a4be330c90d54f subproject-junction.bst:base/alpine.bst 
+      cached c93bc58b2fe8afba05ea49c1a941775a15442a56ea826dee5c017641de7c469c subproject-junction.bst:base.bst 
+   buildable df66a655e4245f317fd4f2784982198759448aea207966923e5d27c3d41082b0 hello.bst 
+===============================================================================
+[--:--:--][df66a655][   build:hello.bst                     ] START   project/hello/df66a655-build.20241106-101709.log
+[--:--:--][df66a655][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][df66a655][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][df66a655][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][df66a655][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][df66a655][   build:hello.bst                     ] START   Staging sources
+[00:00:00][df66a655][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][df66a655][   build:hello.bst                     ] START   Running commands
+
+    export NOCONFIGURE=1;
+    
+    if [ -x ./configure ]; then true;
+    elif [ -x ./autogen ]; then ./autogen;
+    elif [ -x ./autogen.sh ]; then ./autogen.sh;
+    elif [ -x ./bootstrap ]; then ./bootstrap;
+    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
+    else autoreconf -ivf .;
+    fi
+    ./configure --prefix=/opt \
+    --exec-prefix=/opt \
+    --bindir=/opt/bin \
+    --sbindir=/opt/sbin \
+    --sysconfdir=/etc \
+    --datadir=/opt/share \
+    --includedir=/opt/include \
+    --libdir=/opt/lib \
+    --libexecdir=/opt/libexec \
+    --localstatedir=/var \
+    --sharedstatedir=/opt/com \
+    Message contains 23 additional lines
+
+[00:00:03][df66a655][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][df66a655][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][df66a655][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:03][df66a655][   build:hello.bst                     ] SUCCESS project/hello/df66a655-build.20241106-101709.log
+[00:00:04][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 0, skipped 3, failed 0 
+    Build Queue: processed 1, skipped 2, failed 0
+
+

As we can see, this time we’ve built the project into the /opt +system prefix instead of the standard /usr prefix.

+

Let’s just take a step back now and summarize the process which +went into this decision:

+
    +

  • The toplevel project.conf exposes the boolean funky option

    • +

  • The toplevel junction subproject-junction.bst chooses to set the +subproject color to blue when the toplevel project is funky

    • +

  • The subproject include/paths.bst include file decides to set the +prefix to /opt in the case that the subproject is blue

    • +

  • The hello.bst includes the include/paths.bst file, in order +to inherit its path configuration from the subproject

    • +
+
+
+

3.3.3. Running the project in both modes

+ +
+user@host:~/junction-includes$ bst shell hello.bst -- /usr/bin/hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888234.047870    5274 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][75ba510d][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][75ba510d][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][75ba510d][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][75ba510d][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][75ba510d][    main:hello.bst                     ] STATUS  Running command
+
+    /usr/bin/hello
+
+Hello World!
+This is amhello 1.0.
+
+ +
+user@host:~/junction-includes$ bst --option funky True shell hello.bst -- /opt/bin/hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888234.677493    5318 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][df66a655][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][df66a655][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][df66a655][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][df66a655][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][df66a655][    main:hello.bst                     ] STATUS  Running command
+
+    /opt/bin/hello
+
+Hello World!
+This is amhello 1.0.
+
+

As expected, the funky variant of the toplevel project installs +the hello world program in the /opt prefix, and as such we +need to call it from there.

+
+
+
+

3.4. Summary

+

In this chapter we’ve discussed how conditional statements +and include files play an essential role +in the API surface of a project, and help to provide some configurability +while preserving encapsulation of the API which a project exposes.

+

We’ve also gone over the mechanics of how these concepts interact and +presented an example which shows how project options can be used in +a recursive context, and how includes can help not only to share code, +but to provide context to dependent projects about how their subprojects +are configured.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/junctions/junction-workspaces.html b/junctions/junction-workspaces.html new file mode 100644 index 000000000..7b08ed083 --- /dev/null +++ b/junctions/junction-workspaces.html @@ -0,0 +1,216 @@ + + + + + + + + + 2. Workspaces and subprojects — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

2. Workspaces and subprojects

+

When developping a project with junctions and +subprojects, you will probably end up needing to work on the subprojects +as well.

+

Opening workspaces works mostly in the +same way as it does with subprojects as it does for elements directly +in your own project.

+
+

Note

+

This section runs commands on the same example project presented in the +previous section, which is distributed with BuildStream in the +doc/examples/junctions +subdirectory.

+
+
+

2.1. Workspacing a junction

+

Sometimes you need to work on the elements declared in a subproject +directly. As the downstream consumer of a junctioned project, it makes +sense that you might need to work on that project as well in order +to satisfy the needs of your downstream project.

+

You can easily work on your subproject by opening a workspace +on the junction element directly.

+ +
+user@host:~/junctions$ bst workspace open --directory workspace_subproject hello-junction.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888247.814005    6497 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][        ][    main:core activity                 ] STATUS  Creating workspace for element hello-junction.bst
+[--:--:--][6d2e5da0][    main:hello-junction.bst            ] START   Staging sources to /home/user/buildstream/doc/examples/junctions/workspace_subproject
+[--:--:--][        ][    main:hello-junction.bst            ] START   Staging local files into CAS
+[00:00:00][        ][    main:hello-junction.bst            ] SUCCESS Staging local files into CAS
+[00:00:00][6d2e5da0][    main:hello-junction.bst            ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/junctions/workspace_subproject
+[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element hello-junction.bst
+
+

After opening a workspace on the junction element, the open workspace +is used to define the subproject, allowing you to make changes to +how the subproject is built, add new dependencies and configure the +subproject in any way.

+
+
+

2.2. Cross-junction workspaces

+

You can open workspaces for elements in the project refered to by the junction +using the syntax bst open ${junction-name}:{element-name}. In this example,

+ +
+user@host:~/junctions$ bst workspace open --directory workspace_hello hello-junction.bst:hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888248.234063    6526 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[--:--:--][3b986a08][   fetch:hello-junction.bst            ] START   junctions/hello-junction/3b986a08-fetch.20241106-101728.log
+[00:00:00][3b986a08][   fetch:hello-junction.bst            ] SUCCESS junctions/hello-junction/3b986a08-fetch.20241106-101728.log
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][        ][    main:core activity                 ] STATUS  Creating workspace for element hello.bst
+[--:--:--][ffeab612][    main:hello-junction.bst:hello.bst  ] START   Staging sources to /home/user/buildstream/doc/examples/junctions/workspace_hello
+[00:00:00][ffeab612][    main:hello-junction.bst:hello.bst  ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/junctions/workspace_hello
+[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element hello-junction.bst:hello.bst
+
+

This has opened a workspace for the hello.bst element from the autotools project. +This workspace can now be used as normal.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/main_about.html b/main_about.html new file mode 100644 index 000000000..f858becab --- /dev/null +++ b/main_about.html @@ -0,0 +1,213 @@ + + + + + + + + + About — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

About

+https://docs.buildstream.build/master/_static/release.svg + +GitHub Workflow Status + +https://img.shields.io/pypi/v/BuildStream.svg + +
+

What is BuildStream?

+

BuildStream is a powerful software integration tool that allows +developers to automate the integration of software components including operating systems, and to +streamline the software development and production process.

+

Some key capabilities of BuildStream include:

+
    +

  • Defining software stacks in a declarative format: BuildStream allows users to define the steps +required to build and integrate software components, including fetching source code and building +dependencies.

    • +

  • Integrating with version control systems: BuildStream can be configured to fetch source code from +popular source code management solutions such as GitLab, GitHub, BitBucket as well as a range of +non-git technologies.

    • +

  • Supporting a wide range of build technologies: BuildStream supports a wide range of technologies, +including key programming languages like C, C++, Python, Rust and Java, as well as many build tools +including Make, CMake, Meson, distutils, pip and others.

    • +

  • Ability to create outputs in a range of formats: e.g. debian packages, flatpak runtimes, sysroots, +system images, for multiple platforms and chipsets.

    • +

  • Flexible architecture: BuildStream is designed to be flexible and extensible, allowing users to +customize their build and integration processes to meet their specific needs and tooling.

    • +

  • Enabling fast and reliable software delivery: By extensibly use of sandboxing techniques and by +its capability to distribute the build, BuildStream helps teams deliver high-quality software faster.

    • +
+
+
+

Why should I use BuildStream?

+

BuildStream offers the following advantages:

+
    +

  • Declarative build instructions/definitions

    +

    BuildStream provides a flexible and extensible framework for the modelling +of software build pipelines in a declarative YAML format, which allows you to +manipulate filesystem data in a controlled, reproducible sandboxed environment.

    +
    • +

  • Support for developer and integrator workflows

    +

    BuildStream provides traceability and reproducibility for integrators handling +stacks of hundreds/thousands of components, as well as workspace features and +shortcuts to minimise cycle-time for developers.

    +
    • +

  • Fast and predictable

    +

    BuildStream can cache previous builds and track changes to source file content +and build/config commands. BuildStream only rebuilds the things that have changed.

    +
    • +

  • Extensible

    +

    You can extend BuildStream to support your favourite build-system.

    +
    • +

  • Bootstrap toolchains and bootable systems

    +

    BuildStream can create full systems and complete toolchains from scratch, for +a range of ISAs including x86_32, x86_64, ARMv7, ARMv8, MIPS.

    +
    • +
+
+
+

How do I use BuildStream?

+

Please refer to the documentation +for information about installing BuildStream, and about the BuildStream YAML format +and plugin options.

+
+
+

How does BuildStream work?

+

BuildStream operates on a set of YAML files (.bst files), as follows:

+
    +

  • Loads the YAML files which describe the target(s) and all dependencies.

    • +

  • Evaluates the version information and build instructions to calculate a build +graph for the target(s) and all dependencies and unique cache-keys for each +element.

    • +

  • Retrieves previously built elements (artifacts) from a local/remote cache, or +builds the elements in a sandboxed environment using the instructions declared +in the .bst files.

    • +

  • Transforms/configures and/or deploys the resulting target(s) based on the +instructions declared in the .bst files.

    • +
+
+
+

How can I get started?

+

To get started, first install BuildStream by following the installation guide +and then follow our tutorial in the +user guide.

+

We also recommend exploring some existing BuildStream projects:

+ +

If you have any questions please ask on our #buildstream channel in irc.gnome.org

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/main_architecture.html b/main_architecture.html new file mode 100644 index 000000000..b9be4fbc2 --- /dev/null +++ b/main_architecture.html @@ -0,0 +1,178 @@ + + + + + + + + + Architecture — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+ +
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/main_core.html b/main_core.html new file mode 100644 index 000000000..cfdeac6b4 --- /dev/null +++ b/main_core.html @@ -0,0 +1,213 @@ + + + + + + + + + Reference — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Reference

+

This section details the core API reference along with +other more elaborate details about BuildStream internals.

+
+ +
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/main_glossary.html b/main_glossary.html new file mode 100644 index 000000000..f7121459a --- /dev/null +++ b/main_glossary.html @@ -0,0 +1,207 @@ + + + + + + + + + Glossary — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Glossary

+
+
.bst file

The configuration for an Element, represented +in YAML format.

+
+
Artifact

The output collected after building an Element.

+

Artifacts can be built from Sources, or pulled from a +Remote Cache, if available.

+
+
Artifact name

The name of an artifact, which can be used +in various commands to operate directly on artifacts, without requiring +the use of a Project.

+
+
Cache

BuildStream leverages various caching techniques in order to avoid +duplicating work.

+

Depending on context, “Cache” might refer to BuildStream’s local +cache or a Remote Cache.

+
+
Core plugin

A Plugin that is contained in the BuildStream +package. These are built-in and don’t need to be defined in the +project configuration.

+

See plugin documentation for more details on core +plugins.

+
+
Dependency

Elements in a BuildStream project can depend +on other elements from the same project. The element dependent upon is +called a dependency.

+

See Dependencies document for more +details.

+
+
Dependency configuration

Additional custom YAML configuration which is used to define +an Element’s relationship with it’s Dependency.

+

This is supported on limited Element implementations, and +each Element defines what configuration it supports.

+

See the dependency documentation for details +on dependency configuration.

+
+
Element

An atom of a BuildStream project. Projects consist of +zero or more elements.

+

During the build process, BuildStream transforms Sources and Dependencies of an +element into its output. The output is called an +Artifact.

+

Configuration for elements is stored in form of .bst files. See Declaring Elements document +for more details on element configurtion.

+
+
External Plugin

A Plugin that is defined in some package other +than BuildStream.

+

External plugins must be declared in the project configuration.

+

For a list of known external plugin repositories, see +External plugins.

+
+
Junction

A special kind of Element, that allows you to +depend on elements from another project.

+

See Junction reference for details on how to +configure junction elements.

+

See Junction guide for details on how to use +junction elements.

+
+
Local Cache

To avoid duplicating work, BuildStream will cache sources, artifacts, +logs, buildtrees etc. in a local cache directory. If these sources or +artifacts are needed another time, BuildStream will use them from the +cache.

+

See Local cache expiry section of the user +guide for details on how to configure the local cache.

+
+
Plugin

BuildStream Plugins define types of Elements +and Sources. Hence, they come in two distinct +varities - Element Plugins and Source Plugins.

+

BuildStream supports some plugins out of the box. It also has support for third party +plugins.

+
+
Project

A collection of Elements.

+

Elements in a project share some central configuration. See +Project configuration to learn how to configure BuildStream projects.

+
+
Remote Cache

A server setup for sharing BuildStream Sources +and/or Artifacts.

+

See cache server documentation for details on +artifact caches.

+
+
Source

Sources describe the input to the build of an Element.

+

In general, an element can have zero or more sources. But, certain +element plugins may restrict the number of allowed sources.

+

Sources are defined in the Sources section of +Element configuration.

+
+
Subproject

Subprojects are projects which are referred +to by a Junction.

+
+
Workspace

Workspaces allow building one or more elements using a local, and +potentially modified, copy of their sources.

+

See Workspaces guide for more details on +how to use workspaces.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/main_install.html b/main_install.html new file mode 100644 index 000000000..6290eb35e --- /dev/null +++ b/main_install.html @@ -0,0 +1,378 @@ + + + + + + + + + Installing from Source — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Installing from Source

+

This page explains how to build and install this version of BuildStream from +source. For general purpose installation instructions consult the +website.

+

For full install instructions, read on:

+ +
+

Installing Dependencies

+
+

Runtime requirements

+

BuildStream requires the following Python environment to run:

+ +

Some Source plugins require specific tools installed +on the host. Here is a commonly used subset based on the +core source plugins +and buildstream-plugins.

+
    +

  • git (for git sources)

    • +

  • lzip (for .tar.lz support in tar sources)

    • +

  • patch (for patch sources)

    • +
+

Some BuildBox tools used by BuildStream require additional host tools:

+
    +

  • bubblewrap (for buildbox-run-bubblewrap)

    • +

  • fusermount3 (for buildbox-fuse)

    • +
+
+
+

Install-time requirements

+

BuildStream contains Cython code which implies the following extra +dependencies at install-time only:

+
    +

  • C and C++ toolchain

    • +

  • Python development headers

    • +
+

These instructions use pip3 to install necessary PyPI packages. +Packagers and integrators may use a different tool and can ignore +the pip dependency below.

+
+
+

Distribution-specific guides

+

This table gives you a list of packages for specific distros:

+ + + + + + + + + + + + + + + + + + + + + + + +

Distribution

Runtime requires

Install requires

Arch Linux

bubblewrap fuse3 git lzip patch python

gcc python-pip

Debian

bubblewrap fuse3 git lzip patch python3

g++ python3-dev python3-pip

Fedora

bubblewrap fuse3 git lzip patch python3

gcc-c++ python3-devel python3-pip

Ubuntu

bubblewrap fuse3 git lzip patch python3

g++ python3-dev python3-pip

+
+
+
+

Installing BuildBox

+

BuildStream depends on the following tools from +BuildBox:

+
+
    +

  • buildbox-casd (to manage local and remote content-addressed storage)

    • +

  • buildbox-fuse (to check out content from the local CAS)

    • +

  • buildbox-run-bubblewrap (to run element commands in a controlled sandbox)

    • +
+
+

These components can be installed from binaries, or built from source.

+
+

Install binaries

+

Browse the release history of static binaries here.

+

Linux x86-64 users can download the latest statically linked binaries here, +The contents of the tarball should be extracted into a directory in PATH, +e.g., ~/.local/bin.

+
+
+

Build from source

+

Buildbox components can be installed from their git repository. +We recommend installing from the latest release tag.

+

See the <”Installation” section here <https://gitlab.com/BuildGrid/buildbox/buildbox/-/blob/master/README.rst#installation>`_.

+

Buildbox contains many components, BuildStream needs only buildbox-casd, +buildbox-fuse and buildbox-run-bubblewrap, which can be selected by +passing options -DTOOLS=OFF -DCASD=ON -DFUSE=ON -DRUN_BUBBLEWRAP=ON to CMake.

+

Finally, configure buildbox-run-bubblewrap as the default buildbox-run +implementation:

+
ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run
+
+
+
+
+
+

Installing BuildStream from a git checkout

+

First, clone the repository. Please check the existing tags in the git +repository and determine which version you want to install:

+
git clone https://github.com/apache/buildstream.git
+cd buildstream
+git checkout <desired release tag>
+
+
+

We recommend pip as a frontend to the underlying setuptools build +system. The following command will build and install BuildStream into your +user’s homedir in ~/.local, and will attempt to fetch and install any +required PyPI dependencies from the internet at the same time:

+
pip3 install --user .
+
+
+

We do not recommend using Pip’s editable mode +(the -e flag). See this issue for discussion.

+

If you want to stop Pip from fetching missing dependencies, use the +--no-index and --no-deps options.

+

Finally, check that the PATH variable contains the ~/.local/bin directory. +If it doesn’t, you could add this to the end of your Bash configuration ~/.bashrc +and restart Bash:

+
export PATH="${PATH}:${HOME}/.local/bin"
+
+
+
+

Note for packagers

+

Distro packaging standards may recommend a specific installation method +for Python packages. BuildStream can be installed with any build frontend that +supports the PEP517 standard. You are +also welcome to use the underlying +setuptools build backend directly.

+
+
+

Installing in virtual environments

+

You can consider installing BuildStream in a +Virtual Environment if you want +to install multiple versions of BuildStream, or to isolate BuildStream and its +dependencies from other Python packages.

+

Here is how to install BuildStream stable and development snapshot releases in +virtual environments of their own:

+
# Install BuildStream stable in an environment called "venv-bst-stable"
+# (At time of writing, this will be BuildStream 1)
+python3 -m venv venv-bst-stable
+venv-bst-stable/bin/pip install BuildStream
+
+# Install BuildStream latest development snapshot in an environment
+# called "venv-bst-latest"
+# (At time of writing, this will be Buildstream 2)
+python3 -m venv venv-bst-latest
+venv-bst-latest/bin/pip install --pre BuildStream
+
+
+

To start using BuildStream from the desired environment, you will need to +activate it first. Activating it will automatically add bst to your PATH +and set up other necessary environment variables:

+
# Use BuildStream stable from venv-bst-stable
+source venv-bst-stable/bin/activate
+bst --version
+
+# Use BuildStream latest from venv-bst-latest
+source venv-bst-latest/bin/activate
+bst --version
+
+# Once you are done, remember to deactivate the virtual environment
+deactivate
+
+
+

If you do not want to manage your virtual environments manually, you can +consider using pipx.

+
+
+
+

Installing completions

+

BuildStream integrates with Bash and Zsh to provide helpful tab-completion. +These completion scripts require manual installation.

+
+

Bash completions

+

Bash completions are provided by the bst completion script, available online +(src/buildstream/data/bst) +and in your local Git clone at src/buildstream/data/bst.

+

To install for the current user, paste the contents of the completion script +into the file ~/.bash_completion.

+

To install completions system-wide, copy the completion script to the system-wide +bash-completion installation path, which you can discover as follows:

+
pkg-config --variable=completionsdir bash-completion
+
+
+

See the bash-completion FAQ +for more information.

+
+
+

Zsh completions

+

Zsh completions are provided by the _bst completion script, available online +(src/buildstream/data/zsh/_bst) +and in your local Git clone at src/buildstream/data/zsh/_bst.

+

Copy the above file to your Zsh completions location. Here are some instructions +for vanilla Zsh, as well as the Prezto and Oh My Zsh frameworks:

+

Zsh:

+
cp src/buildstream/data/zsh/_bst ~/.zfunc/_bst
+
+
+

You must then add the following lines in your ~/.zshrc, if they do not already exist:

+
fpath+=~/.zfunc
+autoload -Uz compinit && compinit
+
+
+

Prezto:

+
cp src/buildstream/data/zsh/_bst ~/.zprezto/modules/completion/external/src/_bst
+
+
+

You may have to reset your zcompdump cache, if you have one, and then restart your shell:

+
rm ~/.zcompdump ${XDG_CACHE_HOME:-$HOME/.cache}/prezto/zcompdump
+
+
+

Oh My Zsh:

+
mkdir $ZSH_CUSTOM/plugins/bst
+cp src/buildstream/data/zsh/_bst $ZSH_CUSTOM/plugins/bst/_bst
+
+
+

You must then add bst to your plugins array in ~/.zshrc:

+
plugins(
+  bst
+  ...
+)
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/main_porting.html b/main_porting.html new file mode 100644 index 000000000..4b4d1909d --- /dev/null +++ b/main_porting.html @@ -0,0 +1,129 @@ + + + + + + + + + Porting guide — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Porting guide

+

This document focuses on porting your existing BuildStream 1 projects to +using BuildStream 2.

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/main_using.html b/main_using.html new file mode 100644 index 000000000..40874b047 --- /dev/null +++ b/main_using.html @@ -0,0 +1,194 @@ + + + + + + + + + Using — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Using

+

This section includes user facing documentation including tutorials, +guides and information on user preferences and configuration.

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/objects.inv b/objects.inv new file mode 100644 index 000000000..6927d66d1 Binary files /dev/null and b/objects.inv differ diff --git a/porting_command_line.html b/porting_command_line.html new file mode 100644 index 000000000..5d6d84418 --- /dev/null +++ b/porting_command_line.html @@ -0,0 +1,279 @@ + + + + + + + + + Porting command line usage — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Porting command line usage

+

This document outlines breaking changes made to the command line interface +in BuildStream 2.

+
+

bst init

+
    +

  • The global --directory option is no longer observed by bst init, instead +the command accepts an optional target directory argument.

    • +

  • The --format-version option has been removed in favor of the new --min-version option.

    • +
+
+
+

bst build

+
    +

  • Tracking is no longer supported at build time and must be performed separately, this +removes the --track, --track-all, --track-except, --track-cross-junctions +and --track-save options from the command.

    +

    To track your elements in BuildStream 2, use the bst source track +command instead.

    +
    • +

  • The --all option which was used to indicate that all dependencies should be built +regardless of whether they are needed for producing the target elements has been removed +in favor of adding the --deps option.

    +

    To acheive the same functionality, use bst build --deps all ....

    +
    • +
+
+
+

bst show

+
    +

  • The plan value is no longer supported as a value for the --deps option.

    • +

  • Values for the %{state} format have changed

    +
      +

    • junction elements will display junction, as these cannot be built

      • +

    • In the case a cached failed build artifact is found, then failed will be displayed

      • +

    • Due to changes in the scheduler, we may observe changes as to when waiting, buildable, fetch needed +are displayed for a given element.

      • +
    +
    • +
+
+
+

bst fetch

+
    +

  • This command has been removed as a top-level command and now exists as bst source fetch

    • +

  • Tracking is no longer supported at fetch time and must be performed separately, this +removes the --track and --track-cross-junctions options from the command.

    +

    To track your elements in BuildStream 2, use the bst source track +command instead.

    +
    • +

  • The plan value is no longer supported as a value for the --deps option. The default +value for the --deps option is now none.

    • +
+
+
+

bst track

+
    +

  • This command has been removed as a top-level command and now exists as bst source track

    • +
+
+
+

bst pull

+
    +

  • This command has been removed as a top-level command and now exists as bst artifact pull

    • +

  • The --remote option has been removed in favor the --artifact-remote option, which can be +specified multiple times.

    • +

  • The values which can be specified by --artifact-remote options have a new format which +is documented here.

    • +
+
+
+

bst push

+
    +

  • This command has been removed as a top-level command and now exists as bst artifact push

    • +

  • The --remote option has been removed in favor the --artifact-remote option, which can be +specified multiple times.

    • +

  • The values which can be specified by --artifact-remote options have a new format which +is documented here.

    • +
+
+
+

bst checkout

+
    +

  • This command has been removed as a top-level command and now exists as bst artifact checkout

    • +

  • The trailing LOCATION argument has been removed in favor of a --directory option.

    +

    BuildStream 1:

    +
    bst checkout element.bst ~/checkout
+
    +
    +

    BuildStream 2:

    +
    bst artifact checkout --directory ~/checkout element.bst
+
    +
    +
    • +
+
+
+

bst shell

+
    +

  • The --sysroot option has been completely removed.

    +

    This is no longer needed for failed builds as the build tree will be cached in a failed build artifact.

    +
    • +

  • Sources and artifacts required to produce the shell environment will now be downloaded +automatically by default.

    • +
+
+
+

bst workspace open

+
    +

  • The --track option is now removed.

    • +

  • The trailing LOCATION argument has been removed in favor of a --directory option.

    +

    BuildStream 1:

    +
    bst workspace open element.bst ~/workspace
+
    +
    +

    BuildStream 2:

    +
    bst workspace open --directory ~/workspace element.bst
+
    +
    +
    • +
+
+
+

bst workspace reset

+
    +

  • The --track option is now removed.

    • +
+
+
+

bst source-bundle

+

This command has been completely removed, but similar behavior can be achieved +using the bst source checkout command.

+

BuildStream 1:

+
bst source-bundle --directory ~/bundle element.bst
+
+
+

BuildStream 2:

+
bst source checkout \
+    --tar ~/sources.tgz \
+    --compression gz \
+    --include-build-scripts \
+    element.bst
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/porting_project.html b/porting_project.html new file mode 100644 index 000000000..fb8b01b08 --- /dev/null +++ b/porting_project.html @@ -0,0 +1,487 @@ + + + + + + + + + Porting the project format — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Porting the project format

+

This document outlines breaking changes made to the project format in BuildStream 2.

+
+

The project.conf

+

This section outlines breaking changes made to the project.conf format.

+
+

Project name

+

Various features related to junction elements have been added +which allow addressing projects by their project names. For this +reason, it is important to ensure that your project names are appropriately unique.

+
+
+

Project versioning

+

Instead of maintaining a separate version number for the format and for BuildStream releases, +projects now declare the minimum version of BuildStream 2 they depend on.

+

The format-version attribute should be removed from your project.conf (if present) and +the min-version attribute must be added.

+
+
+

Some attributes can only be specified in project.conf

+

Whenever specifying any of the following toplevel project attributes, they must be +specified inside the project.conf itself and cannot be included +from a separate file:

+ +
+
+

Artifact cache configuration

+

The format for declaring artifact caches which are associated to +your project have been completely redesigned.

+

BuildStream 1:

+
#
+# We used to specify a single URL
+#
+artifacts:
+  url: https://foo.com/artifacts
+
+
+

BuildStream 2:

+
#
+# Now we declare a list, and credentials have been split
+# out into a separate "auth" dictionary
+#
+artifacts:
+- url: https://foo.com:11001
+  auth:
+    server-cert: server.crt
+
+
+
+
+

Loading plugins

+

The format for loading plugins has been completely redesigned.

+
+

Tip

+

A new method for loading plugins through junctions +has been added. In the interest of ensuring strong determinism and reliability it is +strongly recommended to use this new method.

+
+
+

Local plugins

+

Here is an example of how loading local plugins has changed.

+

BuildStream 1:

+
plugins:
+
+- origin: local
+  path: plugins/sources
+
+  #
+  # We used to specify version numbers, these no longer exist.
+  #
+  sources:
+    mysource: 0
+
+
+

BuildStream 2:

+
plugins:
+
+- origin: local
+  path: plugins/sources
+
+  #
+  # Now we merely specify a list of plugins to load from
+  # a given project local directory
+  #
+  sources:
+  - mysource
+
+
+
+
+

Pip plugins

+

Here is an example of how loading pip plugins has changed.

+

BuildStream 1:

+
plugins:
+
+- origin: pip
+
+  package-name: vegetables
+
+  #
+  # We used to specify version numbers, these no longer exist.
+  #
+  elements:
+    potato: 0
+
+
+

BuildStream 2:

+
plugins:
+
+- origin: pip
+
+  #
+  # We can now specify version constraints
+  #
+  package-name: vegetables>=1.2
+
+  #
+  # Now we merely specify a list of plugins to load from
+  # a given pip package that is expected to be installed
+  #
+  elements:
+  - potato
+
+
+
+
+
+
+

Core elements

+

This section outlines breaking changes made to core element plugins which +may require you to make changes to your project.

+
+

The stack element

+

Stack elements dependencies are now hard required to be both build and runtime dependencies.

+
+
+

The script element

+

The layout attribute has now been removed in favor of dependency level configuration.

+

Here is an example script which collects a manifest of all files in the hypothetical +system.bst element, using a hypothetical base runtime element base-utilities.bst.

+

BuildStream 1:

+
kind: script
+
+build-depends:
+- base-utilities.bst
+- system.bst
+
+config:
+  #
+  # The old format was redundant and required explicit layout
+  # of the dependencies already declared above.
+  #
+  layout:
+  - element: base-utilities.bst
+    destination: /
+  - element: system.bst
+    destination: "%{build-root}"
+
+  commands:
+  - find %{build-root} > %{install-root}/manifest.log
+
+
+

BuildStream 2:

+
kind: script
+
+#
+# The default location is "/" so there is no need to configure
+# the "base-utilities.bst" dependency
+#
+build-depends:
+- base-utilities.bst
+- system.bst
+  config:
+    location: "%{build-root}"
+
+config:
+  commands:
+  - find %{build-root} > %{install-root}/manifest.log
+
+
+
+

Tip

+

The location dependency level configuration is also supported by all +BuildElement plugins.

+
+
+
+

The junction element

+

The YAML format for declaring junctions has not changed, however the way that +multiple junctions interact in a loaded pipeline has changed significantly.

+

Specifically, the element name used to declare +a junction no longer has any special significance, whereas in BuildStream 1 +the junction’s name is used to coalesce matching junctions in subprojects.

+

BuildStream 2 offers more flexibility in this regard, and allows you to inherit +a junction from a subproject, by using a link element directly +in place of a junction, and/or explicitly override the configuration of a subproject’s +junction using the new overrides configuration attribute which the junction +element now provides.

+

Consult the junction element documentation for a more +detailed explanation.

+
+
+
+

Migrated plugins

+

A majority of the plugins which used to be considered core plugins have been removed +from BuildStream in favor of a more modular and distributed approach. The remaining +core plugins are documented here.

+

Any core plugins which you have been using in BuildStream 1 which have been migrated +to separate repositories will need to be accessed externally.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Plugin

New location

Element plugins

make

buildstream-plugins

autotools

buildstream-plugins

cmake

buildstream-plugins

distutils

buildstream-plugins (as setuptools)

pip

buildstream-plugins

meson

buildstream-plugins

qmake

bst-plugins-experimental

makemaker

bst-plugins-experimental

modulebuild

bst-plugins-experimental

Source plugins

bzr

buildstream-plugins

git

buildstream-plugins

patch

buildstream-plugins

pip

buildstream-plugins

deb

bst-plugins-experimental

ostree

bst-plugins-experimental

zip

bst-plugins-experimental

+
+

Attention

+

YAML composition with externally loaded plugins

+

Note that when YAML composition occurs with plugins loaded +from external projects, the plugin defaults will be composited with your project.conf +and not with the project.conf originating in the external project containing the plugin.

+
+
+

Example of externally loaded plugin

+

It is recommended to transition directly loading these plugins through junctions, +which can be done as follows.

+
+

Create an alias for PyPI in your project.conf

+
aliases:
+  pypi: https://files.pythonhosted.org/packages/
+
+
+
+
+

Create buildstream-plugins-junction.bst

+

Create a junction which accesses the release tarball of the plugin repository.

+
kind: junction
+sources:
+- kind: tar
+  url: pypi:e2/d8/ed9e849a1386297f854f9fa0200f3fa108498c0fdb5c86468c1601c7e571/buildstream-plugins-1.91.0.tar.gz
+  ref: 44c6ea15d15476b68d0767c1d410d416f71544e57be572201058f8b3d3b05f83
+
+
+
+
+

Declare the plugin you want to use in your project.conf

+

This will make the make and meson element plugins from the +buildstream-plugins project available for use in your project.

+
plugins:
+- origin: junction
+  junction: buildstream-plugins-junction.bst
+  elements:
+  - make
+  - meson
+
+
+
+
+
+
+

Miscellaneous

+

Here we list some miscellaneous breaking changes to the format in general.

+
+

Element naming

+

The names of elements have become more restrictive, for example +they must have the .bst extension.

+
+
+

Overlap whitelist

+

The overlap whitelist, which is the public data +found on elements which indicate which files an element can overwrite, must now +be expressed with absolute paths.

+
+
+

Strip commands

+

The default strip-commands which BuildElement implementations +use to split out debug symbols from binaries have been removed.

+

This can be solved by declaring a value for the %{strip-binaries} variable which +will be used for this purpose.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/porting_user_configuration.html b/porting_user_configuration.html new file mode 100644 index 000000000..1c375bfd7 --- /dev/null +++ b/porting_user_configuration.html @@ -0,0 +1,162 @@ + + + + + + + + + Porting the buildstream.conf — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Porting the buildstream.conf

+

This document outlines breaking changes made to the user configuration +in BuildStream 2.

+
+

Filename and parallel installation

+

The default filename to load user configuration remains unchanged, however, +if you plan to install and use both versions of BuildStream on the same +host, it is recommended to keep your BuildStream 2 configuration in a +file named buildstream2.conf.

+
+
+

Working directories

+

The builddir and artifactdir have been removed in favor of the new cachedir.

+
+

BuildStream 1:

+
builddir: ${XDG_CACHE_HOME}/buildstream/build
+artifactdir: ${XDG_CACHE_HOME}/buildstream/artifacts
+
+
+
+
+

BuildStream 2:

+
cachedir: ${XDG_CACHE_HOME}/buildstream/cache
+
+
+
+
+
+

Remote cache configuration

+

The configuration for remote artifact caches has been completely +redesigned, please refer to the artifact cache configuration documentation +for details on how to configure remotes in BuildStream 2.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/py-modindex.html b/py-modindex.html new file mode 100644 index 000000000..b459c29fe --- /dev/null +++ b/py-modindex.html @@ -0,0 +1,302 @@ + + + + + + + + Python Module Index — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    +
    • +
  • Python Module Index
    • +
  • +
    • +
+
+
+
+
+ + +

Python Module Index

+ +
+ b | + d | + e | + n | + p | + s | + t | + u +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
 
+ b
+ buildstream +
+ buildstream.buildelement +
 
+ d
+ buildstream.downloadablefilesource +
 
+ e
+ buildstream.element +
+ buildstream.exceptions +
+ elements +
    + elements.compose +
    + elements.filter +
    + elements.import +
    + elements.junction +
    + elements.link +
    + elements.manual +
    + elements.script +
    + elements.stack +
 
+ n
+ buildstream.node +
 
+ p
+ buildstream.plugin +
 
+ s
+ buildstream.sandbox +
    + buildstream.sandbox.sandbox +
+ buildstream.scriptelement +
+ buildstream.source +
+ buildstream.sourcemirror +
+ buildstream.storage +
    + buildstream.storage.directory +
+ sources +
    + sources.local +
    + sources.remote +
    + sources.tar +
    + sources.workspace +
 
+ t
+ buildstream.types +
 
+ u
+ buildstream.utils +
+ + +
+
+
+ +
+ +
+

© Copyright 2017-2022, The Apache Software Foundation.

+
+ + Built with Sphinx using a + theme + provided by Read the Docs. + + +
+
+
+
+
+ + + + \ No newline at end of file diff --git a/search.html b/search.html new file mode 100644 index 000000000..4ceed6bda --- /dev/null +++ b/search.html @@ -0,0 +1,126 @@ + + + + + + + + Search — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    +
    • +
  • Search
    • +
  • +
    • +
+
+
+
+
+ + + + +
+ +
+ +
+
+
+ +
+ +
+

© Copyright 2017-2022, The Apache Software Foundation.

+
+ + Built with Sphinx using a + theme + provided by Read the Docs. + + +
+
+
+
+
+ + + + + + + + + \ No newline at end of file diff --git a/searchindex.js b/searchindex.js new file mode 100644 index 000000000..fbec39a3d --- /dev/null +++ b/searchindex.js @@ -0,0 +1 @@ +Search.setIndex({"alltitles": {"(!) Assertions": [[44, "assertions"]], "(<) List Prepend": [[44, "list-prepend"]], "(=) List Overwrite": [[44, "list-overwrite"]], "(>) List Append": [[44, "list-append"]], "(?) Conditionals": [[44, "conditionals"]], "(@) Include": [[44, "include"]], ".github/compose/ci.docker-compose.yml": [[54, "github-compose-ci-docker-compose-yml"], [54, "id3"]], ".github/workflows/ci.yml and .github/workflows/release.yml": [[54, "github-workflows-ci-yml-and-github-workflows-release-yml"], [54, "id4"]], "1. Builtin defaults": [[44, "builtin-defaults"]], "1. Set up a directory containing mirrors": [[41, "set-up-a-directory-containing-mirrors"], [42, "set-up-a-directory-containing-mirrors"]], "2. Configure lighttpd": [[41, "configure-lighttpd"], [42, "configure-lighttpd"]], "2. Project configuration": [[44, "project-configuration"]], "3. Plugin defaults": [[44, "plugin-defaults"]], "3. Start lighttpd": [[41, "start-lighttpd"], [42, "start-lighttpd"]], "4. Project configuration overrides": [[44, "project-configuration-overrides"]], "4. Test that you can fetch from it": [[41, "test-that-you-can-fetch-from-it"], [42, "test-that-you-can-fetch-from-it"]], "5. Configure the project to use the mirror": [[41, "configure-the-project-to-use-the-mirror"], [42, "configure-the-project-to-use-the-mirror"]], "5. Plugin declarations": [[44, "plugin-declarations"]], "6. Test that the mirror works": [[41, "test-that-the-mirror-works"], [42, "test-that-the-mirror-works"]], "A simple example": [[62, "a-simple-example"]], "ABI compatibility for binary Python packages": [[54, "abi-compatibility-for-binary-python-packages"]], "About": [[65, null]], "Abstract Methods": [[14, "abstract-methods"], [17, "abstract-methods"], [21, "abstract-methods"], [21, "id1"], [22, "abstract-methods"]], "Abstract method implementations": [[12, "abstract-method-implementations"]], "Abstract methods": [[48, "abstract-methods"]], "Accessing previous sources": [[21, "accessing-previous-sources"]], "Accessors and mutators": [[48, "accessors-and-mutators"]], "Add some content": [[81, "add-some-content"]], "Add tests": [[57, "add-tests"]], "Adding BuildStream command output": [[56, "adding-buildstream-command-output"]], "Adding core plugins": [[57, null]], "Adding new features": [[0, "adding-new-features"]], "Adding support for a new Python release": [[54, "adding-support-for-a-new-python-release"]], "Adding tests": [[55, "adding-tests"]], "Additional writings": [[27, null]], "Addressing elements": [[43, "addressing-elements"]], "Always be consistent": [[48, "always-be-consistent"]], "Approximate PEP-8 Style": [[48, "approximate-pep-8-style"]], "Architecture": [[45, "architecture"], [66, null]], "Artifact cache configuration": [[73, "artifact-cache-configuration"]], "Artifact cache servers": [[85, "artifact-cache-servers"], [85, "id15"]], "Artifact caches": [[3, "artifact-caches"]], "Artifact caches and other storage": [[8, "artifact-caches-and-other-storage"]], "Artifact munging": [[60, "artifact-munging"]], "Artifact names": [[84, "artifact-names"]], "Artifact server": [[45, "artifact-server"]], "Artifact subcommands": [[84, "artifact-subcommands"]], "Attributes": [[84, "attributes"], [85, "attributes"], [85, "id1"], [85, "id2"], [85, "id3"], [85, "id4"], [85, "id5"], [85, "id6"], [85, "id7"], [85, "id8"], [85, "id13"]], "Authentication": [[85, "authentication"]], "Avoid tail calling": [[48, "avoid-tail-calling"]], "Avoid transient state on instances": [[48, "avoid-transient-state-on-instances"]], "Bash completions": [[69, "bash-completions"]], "Basic behavior": [[46, "basic-behavior"]], "Benchmarking framework": [[52, "benchmarking-framework"]], "Bonus: lighttpd conf for git and tar": [[41, "bonus-lighttpd-conf-for-git-and-tar"]], "Boolean": [[45, "boolean"]], "Branch names": [[0, "branch-names"]], "BstError parameters": [[48, "bsterror-parameters"]], "Build Phase": [[14, "build-phase"]], "Build controls": [[85, "build-controls"]], "Build from source": [[69, "build-from-source"]], "Build the element": [[81, "build-the-element"]], "Build the hello.bst element": [[79, "build-the-hello-bst-element"], [82, "build-the-hello-bst-element"], [83, "build-the-hello-bst-element"]], "Build-Depends": [[43, "build-depends"]], "BuildElement - Abstract class for build elements": [[12, null]], "BuildStream 1:": [[74, "buildstream-1"]], "BuildStream 2:": [[74, "buildstream-2"]], "BuildStream and Docker": [[1, null]], "BuildStream documentation": [[61, null]], "BuildStream specific requirements": [[87, "buildstream-specific-requirements"]], "Buildbarn": [[86, "buildbarn"]], "Building Docs": [[56, "building-docs"]], "Building and running": [[62, "building-and-running"]], "Building and running hello-dynamic.bst": [[31, "building-and-running-hello-dynamic-bst"]], "Building and running hello-static.bst": [[31, "building-and-running-hello-static-bst"]], "Building hello.bst element with options": [[80, "building-hello-bst-element-with-options"]], "Building the project": [[58, "building-the-project"]], "Building the project in funky mode": [[63, "building-the-project-in-funky-mode"]], "Building the project normally": [[63, "building-the-project-normally"]], "Built-in functionality": [[12, "built-in-functionality"], [21, "built-in-functionality"]], "Builtin defaults": [[45, "builtin-defaults"]], "Builtin public data": [[47, null]], "Bump cython version": [[54, "bump-cython-version"]], "Cache control": [[85, "cache-control"]], "Cache key types": [[2, "cache-key-types"]], "Cache keys": [[2, null]], "Cache server lists": [[85, "cache-server-lists"]], "Cache servers": [[85, "cache-servers"]], "Caches": [[3, null]], "Checking out and deploying toplevel stacks": [[40, "checking-out-and-deploying-toplevel-stacks"]], "Class Reference": [[14, "class-reference"], [16, "class-reference"], [17, "class-reference"], [21, "class-reference"], [22, "class-reference"]], "Class structure and ordering": [[48, "class-structure-and-ordering"]], "Closing your workspace": [[32, "closing-your-workspace"]], "Coding guidelines": [[48, null]], "Combining projects": [[91, null]], "Command line": [[85, "command-line"], [85, "id9"]], "Commands": [[84, null]], "Commit messages": [[0, "commit-messages"]], "Committer access": [[0, "committer-access"]], "Common properties": [[45, "common-properties"]], "Commonly used parameters": [[84, "commonly-used-parameters"]], "Composition": [[44, "composition"], [58, null]], "Conditional configuration of subproject": [[63, "conditional-configuration-of-subproject"]], "Config": [[43, "config"]], "Configurable Warnings": [[17, "configurable-warnings"], [45, "configurable-warnings"]], "Configuration file": [[85, "configuration-file"]], "Configuring Cache Servers": [[86, null]], "Content Addressable Storage (CAS)": [[3, "content-addressable-storage-cas"]], "Context": [[4, "context"]], "Contributing": [[0, null]], "Contributing to the UI": [[53, null]], "Core elements": [[73, "core-elements"]], "Create an alias for PyPI in your project.conf": [[73, "create-an-alias-for-pypi-in-your-project-conf"]], "Create buildstream-plugins-junction.bst": [[73, "create-buildstream-plugins-junction-bst"]], "Creating and using a git mirror": [[41, null]], "Creating and using a tar mirror": [[42, null]], "Creating the project": [[81, "creating-the-project"]], "Cross-junction dependencies": [[43, "cross-junction-dependencies"]], "Cross-junction workspaces": [[64, "cross-junction-workspaces"]], "Customizing the shell": [[45, "customizing-the-shell"]], "Data model": [[4, null]], "Declare the element": [[81, "declare-the-element"]], "Declare the plugin you want to use in your project.conf": [[73, "declare-the-plugin-you-want-to-use-in-your-project-conf"]], "Declaring and overriding variables": [[43, "declaring-and-overriding-variables"]], "Declaring elements": [[43, null]], "Default configuration": [[85, "default-configuration"]], "Default mirror": [[85, "default-mirror"]], "Default targets": [[45, "default-targets"]], "Dependencies": [[43, "dependencies"]], "Dependency model": [[5, null]], "Dependency types": [[43, "dependency-types"]], "Depends": [[43, "depends"]], "Developing": [[88, null]], "Device access": [[9, "device-access"]], "Directives": [[44, "directives"]], "Directory - Interfacing with files": [[24, null]], "Directory structure": [[44, "directory-structure"]], "Displaying information": [[53, "displaying-information"]], "Distribution-specific guides": [[69, "distribution-specific-guides"]], "Documentation formatting policy": [[56, "documentation-formatting-policy"]], "Documenting Exceptions": [[48, "documenting-exceptions"]], "Documenting symbols": [[48, "documenting-symbols"]], "DownloadableFileSource - Abstract class for sources downloaded from a URI": [[13, null]], "Duplicate junctions": [[45, "duplicate-junctions"]], "Element": [[4, "element"]], "Element - Base element class": [[14, null]], "Element basics": [[43, "element-basics"]], "Element composition": [[4, "element-composition"]], "Element data structure": [[4, "element-data-structure"]], "Element default configuration": [[45, "element-default-configuration"]], "Element mask": [[45, "element-mask"]], "Element names and paths": [[43, "element-names-and-paths"]], "Element naming": [[73, "element-naming"]], "Element naming rules": [[43, "element-naming-rules"]], "Element overrides": [[45, "element-overrides"]], "Element path": [[45, "element-path"]], "Element.assemble()": [[12, "element-assemble"]], "Element.configure_sandbox()": [[12, "element-configure-sandbox"]], "Element.stage()": [[12, "element-stage"]], "Elements": [[30, "elements"]], "Enhancing exceptions": [[48, "enhancing-exceptions"]], "Enumeration": [[45, "enumeration"]], "Environment": [[43, "environment"], [45, "environment"]], "Environment assignments": [[45, "environment-assignments"]], "Error handling": [[48, "error-handling"]], "Essentials": [[45, "essentials"]], "Everything is already built": [[31, "everything-is-already-built"]], "Example of externally loaded plugin": [[73, "example-of-externally-loaded-plugin"]], "Example of split rule declaration": [[58, "example-of-split-rule-declaration"]], "Example working configuration": [[87, "example-working-configuration"]], "Examples": [[89, null]], "Exceptions - API for Error Handling": [[15, null]], "Expressing dependencies": [[43, "expressing-dependencies"]], "Extend the cachekey test": [[57, "extend-the-cachekey-test"]], "External plugins": [[30, "external-plugins"]], "Fetch controls": [[85, "fetch-controls"]], "File naming convention": [[48, "file-naming-convention"]], "Filename and parallel installation": [[74, "filename-and-parallel-installation"]], "Filesystem access": [[9, "filesystem-access"]], "Filesystem metadata": [[9, "filesystem-metadata"]], "Filing issues": [[0, "filing-issues"]], "Filtering": [[59, null]], "Fixing bugs": [[0, "fixing-bugs"]], "Fixing performance issues": [[52, "fixing-performance-issues"]], "Flags": [[45, "flags"]], "Format": [[46, "format"]], "Formatting code": [[55, "formatting-code"]], "Foundation types": [[25, null]], "Further information": [[0, "further-information"]], "Further reading": [[41, "further-reading"], [42, "further-reading"], [62, "further-reading"]], "Generate Docker images": [[1, "generate-docker-images"]], "Generating protocol buffers": [[49, null]], "Generating the unique key": [[21, "generating-the-unique-key"]], "Getting started": [[92, null]], "Global caches": [[85, "global-caches"], [85, "id10"]], "Glossary": [[68, null]], "Guidelines": [[56, "guidelines"]], "Handling files": [[90, null]], "Host files": [[45, "host-files"]], "How can I get started?": [[65, "how-can-i-get-started"]], "How commit access is granted": [[0, "how-commit-access-is-granted"]], "How do I use BuildStream?": [[65, "how-do-i-use-buildstream"]], "How does BuildStream work?": [[65, "how-does-buildstream-work"]], "Implementation of platform properties": [[87, "implementation-of-platform-properties"]], "Imports": [[48, "imports"]], "Includes": [[63, "includes"]], "Including configuration from a subproject": [[63, "including-configuration-from-a-subproject"]], "Incremental builds": [[32, "incremental-builds"]], "Install Location": [[12, "install-location"]], "Install binaries": [[69, "install-binaries"]], "Install-time requirements": [[69, "install-time-requirements"]], "Installing BuildBox": [[69, "installing-buildbox"]], "Installing BuildStream from a git checkout": [[69, "installing-buildstream-from-a-git-checkout"]], "Installing Dependencies": [[69, "installing-dependencies"]], "Installing build dependencies": [[55, "installing-build-dependencies"]], "Installing completions": [[69, "installing-completions"]], "Installing from Source": [[69, null]], "Installing in virtual environments": [[69, "installing-in-virtual-environments"]], "Installing runtime dependencies": [[55, "installing-runtime-dependencies"]], "Instance variables": [[48, "instance-variables"]], "Integration commands": [[47, "integration-commands"], [82, null]], "Interactive shell command": [[45, "interactive-shell-command"]], "Internal junctions": [[45, "internal-junctions"]], "Introducing split rules": [[58, "introducing-split-rules"]], "Introduction": [[9, "introduction"], [44, null]], "Jobs": [[10, "jobs"]], "Junction elements": [[62, null]], "Junction plugins": [[45, "junction-plugins"]], "Junctions": [[45, "junctions"]], "Kind": [[43, "kind"]], "Known implementations": [[86, "known-implementations"]], "Linking to other junctions": [[36, "linking-to-other-junctions"]], "Linux": [[9, "linux"]], "List the content": [[58, "list-the-content"]], "Loading and saving": [[21, "loading-and-saving"]], "Loading plugins": [[45, "loading-plugins"], [73, "loading-plugins"]], "Local plugins": [[45, "local-plugins"], [73, "local-plugins"]], "Location for configuring the project": [[12, "location-for-configuring-the-project"]], "Location for running commands": [[12, "location-for-running-commands"]], "Location for staging dependencies": [[12, "location-for-staging-dependencies"]], "Logging controls": [[85, "logging-controls"]], "Looking at public data": [[82, "looking-at-public-data"]], "Looking at variables": [[79, "looking-at-variables"]], "Making code changes": [[32, "making-code-changes"]], "Making releases": [[50, null]], "Man pages": [[56, "man-pages"]], "Managing data files": [[51, null]], "Managing internal state": [[21, "managing-internal-state"]], "Mapping source aliases of subprojects": [[45, "mapping-source-aliases-of-subprojects"]], "Measuring performance": [[52, null]], "Migrated plugins": [[73, "migrated-plugins"]], "Minimize API surfaces": [[48, "minimize-api-surfaces"]], "Minimize arguments in methods": [[48, "minimize-arguments-in-methods"]], "Minimum version": [[45, "minimum-version"]], "Mirrors": [[45, "mirrors"], [85, "mirrors"]], "Miscellaneous": [[14, "miscellaneous"], [73, "miscellaneous"]], "Mitigating overlapping files": [[60, "mitigating-overlapping-files"]], "Multiple project instances": [[36, "multiple-project-instances"]], "Nested Junctions": [[36, "nested-junctions"]], "Network access": [[9, "network-access"]], "Node - Parsed YAML configuration": [[16, null]], "Node types": [[16, "node-types"]], "Non-strict build plan": [[2, "non-strict-build-plan"]], "Note for packagers": [[69, "note-for-packagers"]], "OS": [[45, "os"]], "Observe the output": [[81, "observe-the-output"]], "Observing coverage": [[55, "observing-coverage"]], "Observing hello-dynamic.bst": [[31, "observing-hello-dynamic-bst"]], "Observing hello-static.bst": [[31, "observing-hello-static-bst"]], "Observing the artifacts": [[59, "observing-the-artifacts"]], "Open a workspace and modify libhello.c": [[31, "open-a-workspace-and-modify-libhello-c"]], "Opening a workspace": [[32, "opening-a-workspace"]], "Optionality and directives": [[80, null]], "Options": [[36, "options"], [45, "options"]], "Organized commits": [[0, "organized-commits"]], "Other POSIX systems": [[9, "other-posix-systems"]], "Overlap whitelist": [[47, "overlap-whitelist"], [73, "overlap-whitelist"]], "Overlapping files": [[60, null]], "Overriding elements": [[36, "overriding-elements"]], "Overriding plugin defaults": [[45, "overriding-plugin-defaults"]], "Overriding subproject junctions": [[36, "overriding-subproject-junctions"]], "Overview": [[31, "overview"], [34, "overview"], [36, "overview"], [37, "overview"], [58, "overview"], [59, "overview"], [60, "overview"], [63, "overview"], [79, "overview"], [80, "overview"], [82, "overview"], [83, "overview"]], "Overview of modules": [[6, null]], "Overview of program flow": [[7, null]], "Paths": [[24, "paths"]], "Pip plugins": [[45, "pip-plugins"], [73, "pip-plugins"]], "Platform notes": [[9, "platform-notes"]], "Plugin - Base plugin class": [[17, null]], "Plugin API reference": [[29, null]], "Plugin Structure": [[17, "plugin-structure"]], "Plugin specific documentation": [[30, null]], "Porting command line usage": [[72, null]], "Porting guide": [[70, null]], "Porting the buildstream.conf": [[74, null]], "Porting the project format": [[73, null]], "Post-release activities": [[50, "post-release-activities"]], "Pre-release changes": [[50, "pre-release-changes"]], "Prerequisites": [[41, "prerequisites"], [42, "prerequisites"]], "Profiling specific parts of BuildStream with BST_PROFILE": [[52, "profiling-specific-parts-of-buildstream-with-bst-profile"]], "Profiling tools": [[52, "profiling-tools"]], "Programming errors": [[48, "programming-errors"]], "Project": [[4, "project"]], "Project configuration": [[45, null]], "Project format": [[28, null], [28, null]], "Project name": [[45, "project-name"], [73, "project-name"]], "Project options": [[63, "project-options"], [63, "id1"], [85, "project-options"]], "Project overrides": [[85, "project-overrides"], [85, "id11"]], "Project recommendations": [[85, "project-recommendations"], [85, "id12"]], "Project specific values": [[85, "project-specific-values"]], "Project structure": [[31, "project-structure"], [58, "project-structure"], [59, "project-structure"], [60, "project-structure"], [63, "project-structure"], [79, "project-structure"], [80, "project-structure"], [82, "project-structure"], [83, "project-structure"]], "Project versioning": [[73, "project-versioning"]], "Public": [[43, "public"]], "Public API surface": [[48, "public-api-surface"]], "Public and private symbols": [[48, "public-and-private-symbols"]], "Pull requests": [[0, "pull-requests"]], "Queue internals": [[10, "queue-internals"]], "Queues": [[10, "queues"]], "Raising exceptions": [[48, "raising-exceptions"]], "Redundant dependency declarations": [[43, "redundant-dependency-declarations"]], "Ref storage": [[45, "ref-storage"]], "Refactor the codebase as needed": [[48, "refactor-the-codebase-as-needed"]], "Reference": [[67, null]], "Referring to variables": [[43, "referring-to-variables"]], "Regenerating code": [[49, "regenerating-code"]], "Regenerating session html": [[56, "regenerating-session-html"]], "Release process": [[50, "release-process"]], "Remote Execution Servers": [[87, null]], "Remote cache configuration": [[74, "remote-cache-configuration"]], "Remote execution": [[8, null], [85, "remote-execution"]], "Remote execution configuration": [[85, "remote-execution-configuration"]], "Remote execution service configuration": [[85, "remote-execution-service-configuration"]], "Remote services": [[85, "remote-services"]], "Remotes": [[84, "remotes"]], "Removing support for a Python release": [[54, "removing-support-for-a-python-release"]], "Requirements": [[50, "requirements"]], "Resources": [[10, "resources"]], "Run BuildStream inside Docker": [[1, "run-buildstream-inside-docker"]], "Run hello.bst built against libhello-filtered.bst": [[59, "run-hello-bst-built-against-libhello-filtered-bst"]], "Run hello.bst built directly against libhello.bst": [[59, "run-hello-bst-built-directly-against-libhello-bst"]], "Run the hello world program": [[79, "run-the-hello-world-program"], [82, "run-the-hello-world-program"], [83, "run-the-hello-world-program"]], "Run the hello world program with options": [[80, "run-the-hello-world-program-with-options"]], "Run the program": [[58, "run-the-program"]], "Running commands": [[83, null]], "Running hello.bst": [[59, "running-hello-bst"]], "Running linters": [[55, "running-linters"]], "Running tests": [[55, "running-tests"]], "Running the project in both modes": [[63, "running-the-project-in-both-modes"]], "Runtime requirements": [[69, "runtime-requirements"]], "Runtime-Depends": [[43, "runtime-depends"]], "Sandbox": [[43, "sandbox"]], "Sandbox - The build sandbox": [[19, null]], "Sandbox extension": [[8, "sandbox-extension"]], "Sandbox options": [[45, "sandbox-options"]], "Sandboxing": [[9, null]], "Scheduler": [[10, null], [10, "id1"]], "Scheduler controls": [[85, "scheduler-controls"]], "Scope": [[5, "scope"]], "ScriptElement - Abstract class for scripting elements": [[20, null]], "Separating source and build directories": [[12, "separating-source-and-build-directories"]], "Setting up a remote cache": [[86, "setting-up-a-remote-cache"]], "Some attributes can only be specified in project.conf": [[73, "some-attributes-can-only-be-specified-in-project-conf"]], "Source": [[4, "source"]], "Source - Base source class": [[21, null]], "Source aliases": [[45, "source-aliases"]], "Source cache server": [[45, "source-cache-server"]], "Source cache servers": [[85, "source-cache-servers"], [85, "id14"]], "Source caches": [[3, "source-caches"]], "Source composition": [[4, "source-composition"]], "Source data structure": [[4, "source-data-structure"]], "Source overrides": [[45, "source-overrides"]], "Source subcommands": [[84, "source-subcommands"]], "SourceFetcher - Object for fetching individual URLs": [[21, "sourcefetcher-object-for-fetching-individual-urls"]], "SourceMirror - Base source mirror class": [[22, null]], "SourceMirror extra data \u201chttp-auth\u201d": [[13, "sourcemirror-extra-data-http-auth"]], "Sources": [[30, "sources"], [36, "sources"], [43, "sources"]], "Split rules": [[45, "split-rules"], [47, "split-rules"]], "Staging the input root as the filesystem root": [[87, "staging-the-input-root-as-the-filesystem-root"]], "Starting a git http server": [[41, "starting-a-git-http-server"]], "Starting a tar server": [[42, "starting-a-tar-server"]], "Strict build plan": [[2, "strict-build-plan"], [85, "strict-build-plan"]], "Strict mode": [[31, null]], "Strip commands": [[73, "strip-commands"]], "Structure": [[2, "structure"]], "Structure of an example": [[56, "structure-of-an-example"]], "Submitting patches": [[0, "submitting-patches"]], "Submodules": [[11, "submodules"], [18, "submodules"], [23, "submodules"]], "Subpackages": [[11, "subpackages"]], "Subproject include/paths.bst": [[63, "subproject-include-paths-bst"]], "Subproject includes": [[63, null]], "Subproject project.conf": [[63, "subproject-project-conf"]], "Summary": [[31, "summary"], [58, "summary"], [59, "summary"], [60, "summary"], [63, "summary"], [79, "summary"], [80, "summary"], [81, "summary"], [82, "summary"], [83, "summary"]], "Suppressing deprecation warnings": [[45, "suppressing-deprecation-warnings"]], "The command line interface": [[53, "the-command-line-interface"]], "The element": [[81, "the-element"]], "The junction element": [[73, "the-junction-element"]], "The project.conf": [[73, "the-project-conf"]], "The project.refs file": [[46, null]], "The script element": [[73, "the-script-element"]], "The source": [[81, "the-source"]], "The stack element": [[73, "the-stack-element"]], "The strip-binaries variable": [[12, "the-strip-binaries-variable"]], "Top-level commands": [[84, "top-level-commands"]], "Toplevel elements/hello.bst": [[63, "toplevel-elements-hello-bst"]], "Toplevel elements/subproject-junction.bst": [[63, "toplevel-elements-subproject-junction-bst"]], "Toplevel project.conf": [[63, "toplevel-project-conf"]], "Track controls": [[85, "track-controls"]], "Tracking new references": [[21, "tracking-new-references"]], "Update documentation index": [[57, "update-documentation-index"]], "Updating BuildStream\u2019s Python dependencies": [[54, null]], "Use of sandboxes outside builds": [[8, "use-of-sandboxes-outside-builds"]], "User and permissions model": [[9, "user-and-permissions-model"]], "User configuration": [[85, null]], "User guide": [[56, "user-guide"]], "Using": [[71, null]], "Using intermediate stacks": [[40, "using-intermediate-stacks"]], "Using the autotools element": [[79, null]], "Using the project": [[31, "using-the-project"], [58, "using-the-project"], [59, "using-the-project"], [60, "using-the-project"], [63, "using-the-project"], [79, "using-the-project"], [80, "using-the-project"], [82, "using-the-project"], [83, "using-the-project"]], "Using the test suite": [[55, null]], "Using toplevel stacks": [[40, "using-toplevel-stacks"]], "Using variables": [[43, "using-variables"]], "Utilities": [[26, null]], "Variables": [[43, "variables"], [45, "variables"]], "Variables declared by BuildStream": [[43, "variables-declared-by-buildstream"]], "Versioning constraints": [[45, "versioning-constraints"]], "Vertical stacking of modules": [[48, "vertical-stacking-of-modules"]], "What elements can and can\u2019t do in the sandbox": [[9, "what-elements-can-and-can-t-do-in-the-sandbox"]], "What is BuildStream?": [[65, "what-is-buildstream"]], "Wheel details": [[54, "wheel-details"]], "Whitelisting overlapping files": [[60, "whitelisting-overlapping-files"]], "Why should I use BuildStream?": [[65, "why-should-i-use-buildstream"]], "Windows CI": [[0, "windows-ci"]], "Working directories": [[74, "working-directories"], [85, "working-directories"]], "Working with the source ref": [[21, "working-with-the-source-ref"]], "Workspace subcommands": [[84, "workspace-subcommands"]], "Workspaces": [[4, "workspaces"], [32, null]], "Workspaces and subprojects": [[64, null]], "Workspacing a junction": [[64, "workspacing-a-junction"]], "Writing documentation": [[56, null]], "Your first project": [[81, null]], "Zsh completions": [[69, "zsh-completions"]], "bst": [[84, "bst"]], "bst artifact": [[84, "bst-artifact"]], "bst artifact checkout": [[84, "bst-artifact-checkout"]], "bst artifact delete": [[84, "bst-artifact-delete"]], "bst artifact list-contents": [[84, "bst-artifact-list-contents"]], "bst artifact log": [[84, "bst-artifact-log"]], "bst artifact pull": [[84, "bst-artifact-pull"]], "bst artifact push": [[84, "bst-artifact-push"]], "bst artifact show": [[84, "bst-artifact-show"]], "bst build": [[72, "bst-build"], [84, "bst-build"]], "bst checkout": [[72, "bst-checkout"]], "bst fetch": [[72, "bst-fetch"]], "bst init": [[72, "bst-init"], [84, "bst-init"]], "bst pull": [[72, "bst-pull"]], "bst push": [[72, "bst-push"]], "bst shell": [[72, "bst-shell"], [84, "bst-shell"]], "bst show": [[72, "bst-show"], [84, "bst-show"]], "bst source": [[84, "bst-source"]], "bst source checkout": [[84, "bst-source-checkout"]], "bst source fetch": [[84, "bst-source-fetch"]], "bst source push": [[84, "bst-source-push"]], "bst source track": [[84, "bst-source-track"]], "bst source-bundle": [[72, "bst-source-bundle"]], "bst track": [[72, "bst-track"]], "bst workspace": [[84, "bst-workspace"]], "bst workspace close": [[84, "bst-workspace-close"]], "bst workspace list": [[84, "bst-workspace-list"]], "bst workspace open": [[72, "bst-workspace-open"], [84, "bst-workspace-open"]], "bst workspace reset": [[72, "bst-workspace-reset"], [84, "bst-workspace-reset"]], "buildstream package": [[11, null]], "buildstream.sandbox package": [[18, null]], "buildstream.storage package": [[23, null]], "compose - Compose the output of multiple elements": [[33, null]], "elements/base.bst": [[83, "elements-base-bst"]], "elements/base/alpine.bst": [[58, "elements-base-alpine-bst"], [82, "elements-base-alpine-bst"], [83, "elements-base-alpine-bst"]], "elements/base/alpine.bst and elements/base.bst": [[79, "elements-base-alpine-bst-and-elements-base-bst"]], "elements/hello-dynamic.bst": [[31, "elements-hello-dynamic-bst"]], "elements/hello-static.bst": [[31, "elements-hello-static-bst"]], "elements/hello.bst": [[59, "elements-hello-bst"], [79, "elements-hello-bst"], [80, "elements-hello-bst"], [81, "elements-hello-bst"], [83, "elements-hello-bst"]], "elements/libhello-filtered.bst": [[59, "elements-libhello-filtered-bst"]], "elements/libhello.bst": [[59, "elements-libhello-bst"]], "elements/libhello.bst and elements/hello.bst": [[82, "elements-libhello-bst-and-elements-hello-bst"]], "elements/runtime-only.bst": [[58, "elements-runtime-only-bst"]], "files/hello/Makefile": [[59, "files-hello-makefile"], [60, "files-hello-makefile"]], "files/hello/Makefile.dynamic": [[31, "files-hello-makefile-dynamic"]], "files/hello/Makefile.static": [[31, "files-hello-makefile-static"]], "files/hello/hello.c": [[59, "files-hello-hello-c"]], "files/libhello/Makefile": [[31, "files-libhello-makefile"], [60, "files-libhello-makefile"]], "files/src/Makefile": [[80, "files-src-makefile"], [83, "files-src-makefile"]], "files/src/hello.c": [[80, "files-src-hello-c"], [83, "files-src-hello-c"]], "filter - Extract a subset of files from another element": [[34, null]], "import - Import sources directly": [[35, null]], "include/greeting.bst": [[80, "include-greeting-bst"]], "junction - Integrate subprojects": [[36, null]], "libhello-filtered.bst": [[59, "libhello-filtered-bst"]], "libhello.bst": [[59, "libhello-bst"]], "link - Link elements": [[37, null]], "local - stage local files and directories": [[75, null]], "manual - Manual build element": [[38, null]], "project.conf": [[59, "project-conf"], [79, "project-conf"], [80, "project-conf"], [81, "project-conf"], [83, "project-conf"]], "project.conf and elements/base.bst": [[82, "project-conf-and-elements-base-bst"]], "pyproject.toml": [[54, "pyproject-toml"], [54, "id2"]], "remote - stage files from remote urls": [[76, null]], "script - Run scripts to create output": [[39, null]], "stack - Symbolic Element for dependency grouping": [[40, null]], "tar - stage files from tar archives": [[77, null]], "tox.ini": [[54, "tox-ini"], [54, "id1"]], "workspace - stage an opened workspace directory": [[78, null]]}, "docnames": ["CONTRIBUTING", "additional_docker", "arch_cachekeys", "arch_caches", "arch_data_model", "arch_dependency_model", "arch_overview", "arch_program_flow", "arch_remote_execution", "arch_sandboxing", "arch_scheduler", "buildstream", "buildstream.buildelement", "buildstream.downloadablefilesource", "buildstream.element", "buildstream.exceptions", "buildstream.node", "buildstream.plugin", "buildstream.sandbox", "buildstream.sandbox.sandbox", "buildstream.scriptelement", "buildstream.source", "buildstream.sourcemirror", "buildstream.storage", "buildstream.storage.directory", "buildstream.types", "buildstream.utils", "core_additional", "core_format", "core_framework", "core_plugins", "developing/strict-mode", "developing/workspaces", "elements/compose", "elements/filter", "elements/import", "elements/junction", "elements/link", "elements/manual", "elements/script", "elements/stack", "examples/git-mirror", "examples/tar-mirror", "format_declaring", "format_intro", "format_project", "format_project_refs", "format_public", "hacking/coding_guidelines", "hacking/grpc_protocols", "hacking/making_releases", "hacking/managing_data_files", "hacking/measuring_performance", "hacking/ui", "hacking/updating_python_deps", "hacking/using_the_testsuite", "hacking/writing_documentation", "hacking/writing_plugins", "handling-files/composition", "handling-files/filtering", "handling-files/overlaps", "index", "junctions/junction-elements", "junctions/junction-includes", "junctions/junction-workspaces", "main_about", "main_architecture", "main_core", "main_glossary", "main_install", "main_porting", "main_using", "porting_command_line", "porting_project", "porting_user_configuration", "sources/local", "sources/remote", "sources/tar", "sources/workspace", "tutorial/autotools", "tutorial/directives", "tutorial/first-project", "tutorial/integration-commands", "tutorial/running-commands", "using_commands", "using_config", "using_configuring_cache_server", "using_configuring_remote_execution", "using_developing", "using_examples", "using_handling_files", "using_junctions", "using_tutorial"], "envversion": {"sphinx": 64, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2}, "filenames": ["CONTRIBUTING.rst", "additional_docker.rst", "arch_cachekeys.rst", "arch_caches.rst", "arch_data_model.rst", "arch_dependency_model.rst", "arch_overview.rst", "arch_program_flow.rst", "arch_remote_execution.rst", "arch_sandboxing.rst", "arch_scheduler.rst", "buildstream.rst", "buildstream.buildelement.rst", "buildstream.downloadablefilesource.rst", "buildstream.element.rst", "buildstream.exceptions.rst", "buildstream.node.rst", "buildstream.plugin.rst", "buildstream.sandbox.rst", "buildstream.sandbox.sandbox.rst", "buildstream.scriptelement.rst", "buildstream.source.rst", "buildstream.sourcemirror.rst", "buildstream.storage.rst", "buildstream.storage.directory.rst", "buildstream.types.rst", "buildstream.utils.rst", "core_additional.rst", "core_format.rst", "core_framework.rst", "core_plugins.rst", "developing/strict-mode.rst", "developing/workspaces.rst", "elements/compose.rst", "elements/filter.rst", "elements/import.rst", "elements/junction.rst", "elements/link.rst", "elements/manual.rst", "elements/script.rst", "elements/stack.rst", "examples/git-mirror.rst", "examples/tar-mirror.rst", "format_declaring.rst", "format_intro.rst", "format_project.rst", "format_project_refs.rst", "format_public.rst", "hacking/coding_guidelines.rst", "hacking/grpc_protocols.rst", "hacking/making_releases.rst", "hacking/managing_data_files.rst", "hacking/measuring_performance.rst", "hacking/ui.rst", "hacking/updating_python_deps.rst", "hacking/using_the_testsuite.rst", "hacking/writing_documentation.rst", "hacking/writing_plugins.rst", "handling-files/composition.rst", "handling-files/filtering.rst", "handling-files/overlaps.rst", "index.rst", "junctions/junction-elements.rst", "junctions/junction-includes.rst", "junctions/junction-workspaces.rst", "main_about.rst", "main_architecture.rst", "main_core.rst", "main_glossary.rst", "main_install.rst", "main_porting.rst", "main_using.rst", "porting_command_line.rst", "porting_project.rst", "porting_user_configuration.rst", "sources/local.rst", "sources/remote.rst", "sources/tar.rst", "sources/workspace.rst", "tutorial/autotools.rst", "tutorial/directives.rst", "tutorial/first-project.rst", "tutorial/integration-commands.rst", "tutorial/running-commands.rst", "using_commands.rst", "using_config.rst", "using_configuring_cache_server.rst", "using_configuring_remote_execution.rst", "using_developing.rst", "using_examples.rst", "using_handling_files.rst", "using_junctions.rst", "using_tutorial.rst"], "indexentries": {"--all": [[84, "cmdoption-bst-workspace-close-a", false], [84, "cmdoption-bst-workspace-reset-a", false]], "--artifact-remote": [[84, "cmdoption-bst-artifact-checkout-artifact-remote", false], [84, "cmdoption-bst-artifact-pull-artifact-remote", false], [84, "cmdoption-bst-artifact-push-artifact-remote", false], [84, "cmdoption-bst-build-artifact-remote", false], [84, "cmdoption-bst-shell-artifact-remote", false]], "--build": [[84, "cmdoption-bst-shell-b", false]], "--builders": [[84, "cmdoption-bst-builders", false]], "--cache-buildtrees": [[84, "cmdoption-bst-cache-buildtrees", false]], "--colors": [[84, "cmdoption-bst-colors", false]], "--compression": [[84, "cmdoption-bst-artifact-checkout-compression", false], [84, "cmdoption-bst-source-checkout-compression", false]], "--config": [[84, "cmdoption-bst-c", false]], "--cross-junctions": [[84, "cmdoption-bst-source-track-J", false]], "--debug": [[84, "cmdoption-bst-debug", false]], "--default-mirror": [[84, "cmdoption-bst-default-mirror", false]], "--deps": [[84, "cmdoption-bst-artifact-checkout-d", false], [84, "cmdoption-bst-artifact-delete-d", false], [84, "cmdoption-bst-artifact-pull-d", false], [84, "cmdoption-bst-artifact-push-d", false], [84, "cmdoption-bst-artifact-show-d", false], [84, "cmdoption-bst-build-d", false], [84, "cmdoption-bst-show-d", false], [84, "cmdoption-bst-source-checkout-d", false], [84, "cmdoption-bst-source-fetch-d", false], [84, "cmdoption-bst-source-push-d", false], [84, "cmdoption-bst-source-track-d", false]], "--directory": [[84, "cmdoption-bst-C", false], [84, "cmdoption-bst-artifact-checkout-directory", false], [84, "cmdoption-bst-source-checkout-directory", false], [84, "cmdoption-bst-workspace-open-directory", false]], "--element-path": [[84, "cmdoption-bst-init-element-path", false]], "--error-lines": [[84, "cmdoption-bst-error-lines", false]], "--except": [[84, "cmdoption-bst-show-except", false], [84, "cmdoption-bst-source-checkout-except", false], [84, "cmdoption-bst-source-fetch-except", false], [84, "cmdoption-bst-source-push-except", false], [84, "cmdoption-bst-source-track-except", false]], "--fetchers": [[84, "cmdoption-bst-fetchers", false]], "--force": [[84, "cmdoption-bst-artifact-checkout-f", false], [84, "cmdoption-bst-init-f", false], [84, "cmdoption-bst-source-checkout-f", false], [84, "cmdoption-bst-workspace-open-f", false]], "--format": [[84, "cmdoption-bst-show-f", false]], "--hardlinks": [[84, "cmdoption-bst-artifact-checkout-hardlinks", false]], "--ignore-project-artifact-remotes": [[84, "cmdoption-bst-artifact-checkout-ignore-project-artifact-remotes", false], [84, "cmdoption-bst-artifact-pull-ignore-project-artifact-remotes", false], [84, "cmdoption-bst-artifact-push-ignore-project-artifact-remotes", false], [84, "cmdoption-bst-build-ignore-project-artifact-remotes", false], [84, "cmdoption-bst-shell-ignore-project-artifact-remotes", false]], "--ignore-project-source-remotes": [[84, "cmdoption-bst-build-ignore-project-source-remotes", false], [84, "cmdoption-bst-shell-ignore-project-source-remotes", false], [84, "cmdoption-bst-source-checkout-ignore-project-source-remotes", false], [84, "cmdoption-bst-source-fetch-ignore-project-source-remotes", false], [84, "cmdoption-bst-source-push-ignore-project-source-remotes", false], [84, "cmdoption-bst-workspace-open-ignore-project-source-remotes", false]], "--include-build-scripts": [[84, "cmdoption-bst-source-checkout-include-build-scripts", false]], "--integrate": [[84, "cmdoption-bst-artifact-checkout-integrate", false]], "--isolate": [[84, "cmdoption-bst-shell-isolate", false]], "--log-file": [[84, "cmdoption-bst-log-file", false]], "--long": [[84, "cmdoption-bst-artifact-list-contents-l", false]], "--max-jobs": [[84, "cmdoption-bst-max-jobs", false]], "--message-lines": [[84, "cmdoption-bst-message-lines", false]], "--min-version": [[84, "cmdoption-bst-init-min-version", false]], "--mount": [[84, "cmdoption-bst-shell-mount", false]], "--network-retries": [[84, "cmdoption-bst-network-retries", false]], "--no-checkout": [[84, "cmdoption-bst-workspace-open-no-checkout", false]], "--no-colors": [[84, "cmdoption-bst-colors", false]], "--no-debug": [[84, "cmdoption-bst-debug", false]], "--no-integrate": [[84, "cmdoption-bst-artifact-checkout-integrate", false]], "--no-interactive": [[84, "cmdoption-bst-no-interactive", false]], "--no-strict": [[84, "cmdoption-bst-strict", false]], "--no-verbose": [[84, "cmdoption-bst-verbose", false]], "--on-error": [[84, "cmdoption-bst-on-error", false]], "--option": [[84, "cmdoption-bst-o", false]], "--order": [[84, "cmdoption-bst-show-order", false]], "--out": [[84, "cmdoption-bst-artifact-log-out", false]], "--project-name": [[84, "cmdoption-bst-init-project-name", false]], "--pull-buildtrees": [[84, "cmdoption-bst-pull-buildtrees", false]], "--pushers": [[84, "cmdoption-bst-pushers", false]], "--remove-dir": [[84, "cmdoption-bst-workspace-close-remove-dir", false]], "--retry-failed": [[84, "cmdoption-bst-build-r", false]], "--soft": [[84, "cmdoption-bst-workspace-reset-soft", false]], "--source-remote": [[84, "cmdoption-bst-build-source-remote", false], [84, "cmdoption-bst-shell-source-remote", false], [84, "cmdoption-bst-source-checkout-source-remote", false], [84, "cmdoption-bst-source-fetch-source-remote", false], [84, "cmdoption-bst-source-push-source-remote", false], [84, "cmdoption-bst-workspace-open-source-remote", false]], "--strict": [[84, "cmdoption-bst-strict", false]], "--tar": [[84, "cmdoption-bst-artifact-checkout-tar", false], [84, "cmdoption-bst-source-checkout-tar", false]], "--use-buildtree": [[84, "cmdoption-bst-shell-t", false]], "--verbose": [[84, "cmdoption-bst-verbose", false]], "--version": [[84, "cmdoption-bst-version", false]], "-a": [[84, "cmdoption-bst-workspace-close-a", false], [84, "cmdoption-bst-workspace-reset-a", false]], "-b": [[84, "cmdoption-bst-shell-b", false]], "-c": [[84, "cmdoption-bst-C", false], [84, "cmdoption-bst-c", false]], "-d": [[84, "cmdoption-bst-artifact-checkout-d", false], [84, "cmdoption-bst-artifact-delete-d", false], [84, "cmdoption-bst-artifact-pull-d", false], [84, "cmdoption-bst-artifact-push-d", false], [84, "cmdoption-bst-artifact-show-d", false], [84, "cmdoption-bst-build-d", false], [84, "cmdoption-bst-show-d", false], [84, "cmdoption-bst-source-checkout-d", false], [84, "cmdoption-bst-source-fetch-d", false], [84, "cmdoption-bst-source-push-d", false], [84, "cmdoption-bst-source-track-d", false]], "-f": [[84, "cmdoption-bst-artifact-checkout-f", false], [84, "cmdoption-bst-init-f", false], [84, "cmdoption-bst-show-f", false], [84, "cmdoption-bst-source-checkout-f", false], [84, "cmdoption-bst-workspace-open-f", false]], "-j": [[84, "cmdoption-bst-source-track-J", false]], "-l": [[84, "cmdoption-bst-artifact-list-contents-l", false]], "-o": [[84, "cmdoption-bst-o", false]], "-r": [[84, "cmdoption-bst-build-r", false]], "-t": [[84, "cmdoption-bst-shell-t", false]], ".bst file": [[68, "term-.bst-file", true]], "add_commands() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.add_commands", false]], "aliassubstitution (class in buildstream.source)": [[21, "buildstream.source.AliasSubstitution", false]], "app (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.APP", false]], "append() (sequencenode method)": [[16, "buildstream.node.SequenceNode.append", false]], "artifact": [[68, "term-Artifact", true]], "artifact (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.ARTIFACT", false]], "artifact name": [[68, "term-Artifact-name", true]], "artifacts": [[84, "cmdoption-bst-artifact-delete-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-list-contents-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-log-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-pull-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-push-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-show-arg-ARTIFACTS", false]], "as_bool() (scalarnode method)": [[16, "buildstream.node.ScalarNode.as_bool", false]], "as_enum() (scalarnode method)": [[16, "buildstream.node.ScalarNode.as_enum", false]], "as_int() (scalarnode method)": [[16, "buildstream.node.ScalarNode.as_int", false]], "as_str() (scalarnode method)": [[16, "buildstream.node.ScalarNode.as_str", false]], "as_str_list() (sequencenode method)": [[16, "buildstream.node.SequenceNode.as_str_list", false]], "assemble() (buildelement method)": [[12, "buildstream.buildelement.BuildElement.assemble", false]], "assemble() (element method)": [[14, "buildstream.element.Element.assemble", false]], "assemble() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.assemble", false]], "bad_characters_in_name (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.BAD_CHARACTERS_IN_NAME", false]], "bad_element_suffix (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.BAD_ELEMENT_SUFFIX", false]], "batch() (sandbox method)": [[19, "buildstream.sandbox.sandbox.Sandbox.batch", false]], "blocking_activity() (plugin method)": [[17, "buildstream.plugin.Plugin.blocking_activity", false]], "bst command line option": [[84, "cmdoption-bst-C", false], [84, "cmdoption-bst-builders", false], [84, "cmdoption-bst-c", false], [84, "cmdoption-bst-cache-buildtrees", false], [84, "cmdoption-bst-colors", false], [84, "cmdoption-bst-debug", false], [84, "cmdoption-bst-default-mirror", false], [84, "cmdoption-bst-error-lines", false], [84, "cmdoption-bst-fetchers", false], [84, "cmdoption-bst-log-file", false], [84, "cmdoption-bst-max-jobs", false], [84, "cmdoption-bst-message-lines", false], [84, "cmdoption-bst-network-retries", false], [84, "cmdoption-bst-no-interactive", false], [84, "cmdoption-bst-o", false], [84, "cmdoption-bst-on-error", false], [84, "cmdoption-bst-pull-buildtrees", false], [84, "cmdoption-bst-pushers", false], [84, "cmdoption-bst-strict", false], [84, "cmdoption-bst-verbose", false], [84, "cmdoption-bst-version", false]], "bst-artifact-checkout command line option": [[84, "cmdoption-bst-artifact-checkout-arg-TARGET", false], [84, "cmdoption-bst-artifact-checkout-artifact-remote", false], [84, "cmdoption-bst-artifact-checkout-compression", false], [84, "cmdoption-bst-artifact-checkout-d", false], [84, "cmdoption-bst-artifact-checkout-directory", false], [84, "cmdoption-bst-artifact-checkout-f", false], [84, "cmdoption-bst-artifact-checkout-hardlinks", false], [84, "cmdoption-bst-artifact-checkout-ignore-project-artifact-remotes", false], [84, "cmdoption-bst-artifact-checkout-integrate", false], [84, "cmdoption-bst-artifact-checkout-tar", false]], "bst-artifact-delete command line option": [[84, "cmdoption-bst-artifact-delete-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-delete-d", false]], "bst-artifact-list-contents command line option": [[84, "cmdoption-bst-artifact-list-contents-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-list-contents-l", false]], "bst-artifact-log command line option": [[84, "cmdoption-bst-artifact-log-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-log-out", false]], "bst-artifact-pull command line option": [[84, "cmdoption-bst-artifact-pull-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-pull-artifact-remote", false], [84, "cmdoption-bst-artifact-pull-d", false], [84, "cmdoption-bst-artifact-pull-ignore-project-artifact-remotes", false]], "bst-artifact-push command line option": [[84, "cmdoption-bst-artifact-push-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-push-artifact-remote", false], [84, "cmdoption-bst-artifact-push-d", false], [84, "cmdoption-bst-artifact-push-ignore-project-artifact-remotes", false]], "bst-artifact-show command line option": [[84, "cmdoption-bst-artifact-show-arg-ARTIFACTS", false], [84, "cmdoption-bst-artifact-show-d", false]], "bst-build command line option": [[84, "cmdoption-bst-build-arg-ELEMENTS", false], [84, "cmdoption-bst-build-artifact-remote", false], [84, "cmdoption-bst-build-d", false], [84, "cmdoption-bst-build-ignore-project-artifact-remotes", false], [84, "cmdoption-bst-build-ignore-project-source-remotes", false], [84, "cmdoption-bst-build-r", false], [84, "cmdoption-bst-build-source-remote", false]], "bst-init command line option": [[84, "cmdoption-bst-init-arg-TARGET_DIRECTORY", false], [84, "cmdoption-bst-init-element-path", false], [84, "cmdoption-bst-init-f", false], [84, "cmdoption-bst-init-min-version", false], [84, "cmdoption-bst-init-project-name", false]], "bst-shell command line option": [[84, "cmdoption-bst-shell-arg-COMMAND", false], [84, "cmdoption-bst-shell-arg-TARGET", false], [84, "cmdoption-bst-shell-artifact-remote", false], [84, "cmdoption-bst-shell-b", false], [84, "cmdoption-bst-shell-ignore-project-artifact-remotes", false], [84, "cmdoption-bst-shell-ignore-project-source-remotes", false], [84, "cmdoption-bst-shell-isolate", false], [84, "cmdoption-bst-shell-mount", false], [84, "cmdoption-bst-shell-source-remote", false], [84, "cmdoption-bst-shell-t", false]], "bst-show command line option": [[84, "cmdoption-bst-show-arg-ELEMENTS", false], [84, "cmdoption-bst-show-d", false], [84, "cmdoption-bst-show-except", false], [84, "cmdoption-bst-show-f", false], [84, "cmdoption-bst-show-order", false]], "bst-source-checkout command line option": [[84, "cmdoption-bst-source-checkout-arg-ELEMENT", false], [84, "cmdoption-bst-source-checkout-compression", false], [84, "cmdoption-bst-source-checkout-d", false], [84, "cmdoption-bst-source-checkout-directory", false], [84, "cmdoption-bst-source-checkout-except", false], [84, "cmdoption-bst-source-checkout-f", false], [84, "cmdoption-bst-source-checkout-ignore-project-source-remotes", false], [84, "cmdoption-bst-source-checkout-include-build-scripts", false], [84, "cmdoption-bst-source-checkout-source-remote", false], [84, "cmdoption-bst-source-checkout-tar", false]], "bst-source-fetch command line option": [[84, "cmdoption-bst-source-fetch-arg-ELEMENTS", false], [84, "cmdoption-bst-source-fetch-d", false], [84, "cmdoption-bst-source-fetch-except", false], [84, "cmdoption-bst-source-fetch-ignore-project-source-remotes", false], [84, "cmdoption-bst-source-fetch-source-remote", false]], "bst-source-push command line option": [[84, "cmdoption-bst-source-push-arg-ELEMENTS", false], [84, "cmdoption-bst-source-push-d", false], [84, "cmdoption-bst-source-push-except", false], [84, "cmdoption-bst-source-push-ignore-project-source-remotes", false], [84, "cmdoption-bst-source-push-source-remote", false]], "bst-source-track command line option": [[84, "cmdoption-bst-source-track-J", false], [84, "cmdoption-bst-source-track-arg-ELEMENTS", false], [84, "cmdoption-bst-source-track-d", false], [84, "cmdoption-bst-source-track-except", false]], "bst-workspace-close command line option": [[84, "cmdoption-bst-workspace-close-a", false], [84, "cmdoption-bst-workspace-close-arg-ELEMENTS", false], [84, "cmdoption-bst-workspace-close-remove-dir", false]], "bst-workspace-open command line option": [[84, "cmdoption-bst-workspace-open-arg-ELEMENTS", false], [84, "cmdoption-bst-workspace-open-directory", false], [84, "cmdoption-bst-workspace-open-f", false], [84, "cmdoption-bst-workspace-open-ignore-project-source-remotes", false], [84, "cmdoption-bst-workspace-open-no-checkout", false], [84, "cmdoption-bst-workspace-open-source-remote", false]], "bst-workspace-reset command line option": [[84, "cmdoption-bst-workspace-reset-a", false], [84, "cmdoption-bst-workspace-reset-arg-ELEMENTS", false], [84, "cmdoption-bst-workspace-reset-soft", false]], "bst_artifact_version (element attribute)": [[14, "buildstream.element.Element.BST_ARTIFACT_VERSION", false]], "bst_element_has_artifact (element attribute)": [[14, "buildstream.element.Element.BST_ELEMENT_HAS_ARTIFACT", false]], "bst_forbid_bdepends (element attribute)": [[14, "buildstream.element.Element.BST_FORBID_BDEPENDS", false]], "bst_forbid_rdepends (element attribute)": [[14, "buildstream.element.Element.BST_FORBID_RDEPENDS", false]], "bst_forbid_rdepends (scriptelement attribute)": [[20, "buildstream.scriptelement.ScriptElement.BST_FORBID_RDEPENDS", false]], "bst_forbid_sources (element attribute)": [[14, "buildstream.element.Element.BST_FORBID_SOURCES", false]], "bst_forbid_sources (scriptelement attribute)": [[20, "buildstream.scriptelement.ScriptElement.BST_FORBID_SOURCES", false]], "bst_min_version (plugin attribute)": [[17, "buildstream.plugin.Plugin.BST_MIN_VERSION", false]], "bst_min_version (sourcemirror attribute)": [[22, "buildstream.sourcemirror.SourceMirror.BST_MIN_VERSION", false]], "bst_plugin_deprecated (plugin attribute)": [[17, "buildstream.plugin.Plugin.BST_PLUGIN_DEPRECATED", false]], "bst_plugin_deprecation_message (plugin attribute)": [[17, "buildstream.plugin.Plugin.BST_PLUGIN_DEPRECATION_MESSAGE", false]], "bst_requires_previous_sources_fetch (source attribute)": [[21, "buildstream.source.Source.BST_REQUIRES_PREVIOUS_SOURCES_FETCH", false]], "bst_requires_previous_sources_stage (source attribute)": [[21, "buildstream.source.Source.BST_REQUIRES_PREVIOUS_SOURCES_STAGE", false]], "bst_requires_previous_sources_track (source attribute)": [[21, "buildstream.source.Source.BST_REQUIRES_PREVIOUS_SOURCES_TRACK", false]], "bst_run_commands (element attribute)": [[14, "buildstream.element.Element.BST_RUN_COMMANDS", false]], "bst_stage_virtual_directory (source attribute)": [[21, "buildstream.source.Source.BST_STAGE_VIRTUAL_DIRECTORY", false]], "bst_strict_rebuild (element attribute)": [[14, "buildstream.element.Element.BST_STRICT_REBUILD", false]], "bst_strict_rebuild (scriptelement attribute)": [[20, "buildstream.scriptelement.ScriptElement.BST_STRICT_REBUILD", false]], "buildelement (class in buildstream.buildelement)": [[12, "buildstream.buildelement.BuildElement", false]], "buildstream": [[11, "module-buildstream", false]], "buildstream.buildelement": [[12, "module-buildstream.buildelement", false]], "buildstream.downloadablefilesource": [[13, "module-buildstream.downloadablefilesource", false]], "buildstream.element": [[14, "module-buildstream.element", false]], "buildstream.exceptions": [[15, "module-buildstream.exceptions", false]], "buildstream.node": [[16, "module-buildstream.node", false]], "buildstream.plugin": [[17, "module-buildstream.plugin", false]], "buildstream.sandbox": [[18, "module-buildstream.sandbox", false]], "buildstream.sandbox.sandbox": [[19, "module-buildstream.sandbox.sandbox", false]], "buildstream.scriptelement": [[20, "module-buildstream.scriptelement", false]], "buildstream.source": [[21, "module-buildstream.source", false]], "buildstream.sourcemirror": [[22, "module-buildstream.sourcemirror", false]], "buildstream.storage": [[23, "module-buildstream.storage", false]], "buildstream.storage.directory": [[24, "module-buildstream.storage.directory", false]], "buildstream.types": [[25, "module-buildstream.types", false]], "buildstream.utils": [[26, "module-buildstream.utils", false]], "cache": [[68, "term-Cache", true]], "call() (plugin method)": [[17, "buildstream.plugin.Plugin.call", false]], "cas (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.CAS", false]], "check_output() (plugin method)": [[17, "buildstream.plugin.Plugin.check_output", false]], "circular_dependency (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.CIRCULAR_DEPENDENCY", false]], "circular_reference (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.CIRCULAR_REFERENCE", false]], "circular_reference_variable (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.CIRCULAR_REFERENCE_VARIABLE", false]], "clone() (mappingnode method)": [[16, "buildstream.node.MappingNode.clone", false]], "clone() (node method)": [[16, "buildstream.node.Node.clone", false]], "clone() (scalarnode method)": [[16, "buildstream.node.ScalarNode.clone", false]], "clone() (sequencenode method)": [[16, "buildstream.node.SequenceNode.clone", false]], "command": [[84, "cmdoption-bst-shell-arg-COMMAND", false]], "common_config_keys (downloadablefilesource attribute)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.COMMON_CONFIG_KEYS", false]], "common_config_keys (source attribute)": [[21, "buildstream.source.Source.COMMON_CONFIG_KEYS", false]], "compute_manifest() (element method)": [[14, "buildstream.element.Element.compute_manifest", false]], "config (dependencyconfiguration attribute)": [[14, "buildstream.element.DependencyConfiguration.config", false]], "configure() (buildelement method)": [[12, "buildstream.buildelement.BuildElement.configure", false]], "configure() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.configure", false]], "configure() (plugin method)": [[17, "buildstream.plugin.Plugin.configure", false]], "configure_dependencies() (buildelement method)": [[12, "buildstream.buildelement.BuildElement.configure_dependencies", false]], "configure_dependencies() (element method)": [[14, "buildstream.element.Element.configure_dependencies", false]], "configure_sandbox() (buildelement method)": [[12, "buildstream.buildelement.BuildElement.configure_sandbox", false]], "configure_sandbox() (element method)": [[14, "buildstream.element.Element.configure_sandbox", false]], "configure_sandbox() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.configure_sandbox", false]], "conflicting_junction (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.CONFLICTING_JUNCTION", false]], "copy_files() (in module buildstream.utils)": [[26, "buildstream.utils.copy_files", false]], "core plugin": [[68, "term-Core-plugin", true]], "corewarnings (class in buildstream.types)": [[25, "buildstream.types.CoreWarnings", false]], "debug() (plugin method)": [[17, "buildstream.plugin.Plugin.debug", false]], "dependencies() (element method)": [[14, "buildstream.element.Element.dependencies", false]], "dependency": [[68, "term-Dependency", true]], "dependency configuration": [[68, "term-Dependency-configuration", true]], "dependencyconfiguration (class in buildstream.element)": [[14, "buildstream.element.DependencyConfiguration", false]], "directory (class in buildstream.storage.directory)": [[24, "buildstream.storage.directory.Directory", false]], "directory (filetype attribute)": [[24, "buildstream.storage.directory.FileType.DIRECTORY", false]], "directoryerror": [[24, "buildstream.storage.directory.DirectoryError", false]], "directoryexistserror": [[26, "buildstream.utils.DirectoryExistsError", false]], "downloadablefilesource (class in buildstream.downloadablefilesource)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource", false]], "element": [[68, "term-Element", true], [84, "cmdoption-bst-source-checkout-arg-ELEMENT", false]], "element (class in buildstream.element)": [[14, "buildstream.element.Element", false]], "element (dependencyconfiguration attribute)": [[14, "buildstream.element.DependencyConfiguration.element", false]], "element (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.ELEMENT", false]], "elementerror": [[14, "buildstream.element.ElementError", false]], "elements": [[84, "cmdoption-bst-build-arg-ELEMENTS", false], [84, "cmdoption-bst-show-arg-ELEMENTS", false], [84, "cmdoption-bst-source-fetch-arg-ELEMENTS", false], [84, "cmdoption-bst-source-push-arg-ELEMENTS", false], [84, "cmdoption-bst-source-track-arg-ELEMENTS", false], [84, "cmdoption-bst-workspace-close-arg-ELEMENTS", false], [84, "cmdoption-bst-workspace-open-arg-ELEMENTS", false], [84, "cmdoption-bst-workspace-reset-arg-ELEMENTS", false]], "elements.compose": [[33, "module-elements.compose", false]], "elements.filter": [[34, "module-elements.filter", false]], "elements.import": [[35, "module-elements.import", false]], "elements.junction": [[36, "module-elements.junction", false]], "elements.link": [[37, "module-elements.link", false]], "elements.manual": [[38, "module-elements.manual", false]], "elements.script": [[39, "module-elements.script", false]], "elements.stack": [[40, "module-elements.stack", false]], "error (overlapaction attribute)": [[25, "buildstream.types.OverlapAction.ERROR", false]], "errordomain (class in buildstream.exceptions)": [[15, "buildstream.exceptions.ErrorDomain", false]], "executable (filestat attribute)": [[24, "buildstream.storage.directory.FileStat.executable", false]], "exists() (directory method)": [[24, "buildstream.storage.directory.Directory.exists", false]], "export_to_tar() (directory method)": [[24, "buildstream.storage.directory.Directory.export_to_tar", false]], "expression_failed (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.EXPRESSION_FAILED", false]], "external plugin": [[68, "term-External-Plugin", true]], "failed_attributes (filelistresult attribute)": [[26, "buildstream.utils.FileListResult.failed_attributes", false]], "fastenum (class in buildstream.types)": [[25, "buildstream.types.FastEnum", false]], "fetch() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.fetch", false]], "fetch() (source method)": [[21, "buildstream.source.Source.fetch", false]], "fetch() (sourcefetcher method)": [[21, "buildstream.source.SourceFetcher.fetch", false]], "file_digest() (directory method)": [[24, "buildstream.storage.directory.Directory.file_digest", false]], "file_type (filestat attribute)": [[24, "buildstream.storage.directory.FileStat.file_type", false]], "filelistresult (class in buildstream.utils)": [[26, "buildstream.utils.FileListResult", false]], "files_written (filelistresult attribute)": [[26, "buildstream.utils.FileListResult.files_written", false]], "filestat (class in buildstream.storage.directory)": [[24, "buildstream.storage.directory.FileStat", false]], "filetype (class in buildstream.storage.directory)": [[24, "buildstream.storage.directory.FileType", false]], "from_dict() (node class method)": [[16, "buildstream.node.Node.from_dict", false]], "generate_script() (buildelement method)": [[12, "buildstream.buildelement.BuildElement.generate_script", false]], "generate_script() (element method)": [[14, "buildstream.element.Element.generate_script", false]], "get_artifact_name() (element method)": [[14, "buildstream.element.Element.get_artifact_name", false]], "get_bool() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_bool", false]], "get_bst_version() (in module buildstream.utils)": [[26, "buildstream.utils.get_bst_version", false]], "get_enum() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_enum", false]], "get_environment() (element method)": [[14, "buildstream.element.Element.get_environment", false]], "get_host_tool() (in module buildstream.utils)": [[26, "buildstream.utils.get_host_tool", false]], "get_int() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_int", false]], "get_kind() (plugin method)": [[17, "buildstream.plugin.Plugin.get_kind", false]], "get_mapping() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_mapping", false]], "get_mirror_directory() (source method)": [[21, "buildstream.source.Source.get_mirror_directory", false]], "get_node() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_node", false]], "get_project_directory() (source method)": [[21, "buildstream.source.Source.get_project_directory", false]], "get_provenance() (node method)": [[16, "buildstream.node.Node.get_provenance", false]], "get_public_data() (element method)": [[14, "buildstream.element.Element.get_public_data", false]], "get_ref() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.get_ref", false]], "get_ref() (source method)": [[21, "buildstream.source.Source.get_ref", false]], "get_scalar() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_scalar", false]], "get_sequence() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_sequence", false]], "get_source_fetchers() (source method)": [[21, "buildstream.source.Source.get_source_fetchers", false]], "get_str() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_str", false]], "get_str_list() (mappingnode method)": [[16, "buildstream.node.MappingNode.get_str_list", false]], "get_umask() (in module buildstream.utils)": [[26, "buildstream.utils.get_umask", false]], "get_unique_key() (buildelement method)": [[12, "buildstream.buildelement.BuildElement.get_unique_key", false]], "get_unique_key() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.get_unique_key", false]], "get_unique_key() (plugin method)": [[17, "buildstream.plugin.Plugin.get_unique_key", false]], "get_unique_key() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.get_unique_key", false]], "get_variable() (element method)": [[14, "buildstream.element.Element.get_variable", false]], "get_virtual_directory() (sandbox method)": [[19, "buildstream.sandbox.sandbox.Sandbox.get_virtual_directory", false]], "glob() (in module buildstream.utils)": [[26, "buildstream.utils.glob", false]], "ignore (overlapaction attribute)": [[25, "buildstream.types.OverlapAction.IGNORE", false]], "ignored (filelistresult attribute)": [[26, "buildstream.utils.FileListResult.ignored", false]], "illegal_composite (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.ILLEGAL_COMPOSITE", false]], "impl (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.IMPL", false]], "import_files() (directory method)": [[24, "buildstream.storage.directory.Directory.import_files", false]], "import_single_file() (directory method)": [[24, "buildstream.storage.directory.Directory.import_single_file", false]], "info() (plugin method)": [[17, "buildstream.plugin.Plugin.info", false]], "init_workspace() (source method)": [[21, "buildstream.source.Source.init_workspace", false]], "init_workspace_directory() (source method)": [[21, "buildstream.source.Source.init_workspace_directory", false]], "integrate() (element method)": [[14, "buildstream.element.Element.integrate", false]], "invalid_data (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.INVALID_DATA", false]], "invalid_dependency_config (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.INVALID_DEPENDENCY_CONFIG", false]], "invalid_junction (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.INVALID_JUNCTION", false]], "invalid_symbol_name (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.INVALID_SYMBOL_NAME", false]], "invalid_yaml (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.INVALID_YAML", false]], "is_cached() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.is_cached", false]], "is_cached() (source method)": [[21, "buildstream.source.Source.is_cached", false]], "is_none() (scalarnode method)": [[16, "buildstream.node.ScalarNode.is_none", false]], "is_resolved() (source method)": [[21, "buildstream.source.Source.is_resolved", false]], "isdir() (directory method)": [[24, "buildstream.storage.directory.Directory.isdir", false]], "isfile() (directory method)": [[24, "buildstream.storage.directory.Directory.isfile", false]], "islink() (directory method)": [[24, "buildstream.storage.directory.Directory.islink", false]], "items() (mappingnode method)": [[16, "buildstream.node.MappingNode.items", false]], "junction": [[68, "term-Junction", true]], "keys() (mappingnode method)": [[16, "buildstream.node.MappingNode.keys", false]], "layout_add() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.layout_add", false]], "link_files() (in module buildstream.utils)": [[26, "buildstream.utils.link_files", false]], "link_forbidden_dependencies (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.LINK_FORBIDDEN_DEPENDENCIES", false]], "list_relative_paths() (directory method)": [[24, "buildstream.storage.directory.Directory.list_relative_paths", false]], "list_relative_paths() (in module buildstream.utils)": [[26, "buildstream.utils.list_relative_paths", false]], "load (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.LOAD", false]], "load_ref() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.load_ref", false]], "load_ref() (source method)": [[21, "buildstream.source.Source.load_ref", false]], "loaderrorreason (class in buildstream.exceptions)": [[15, "buildstream.exceptions.LoadErrorReason", false]], "loading_directory (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.LOADING_DIRECTORY", false]], "local cache": [[68, "term-Local-Cache", true]], "log() (plugin method)": [[17, "buildstream.plugin.Plugin.log", false]], "mapping_at() (sequencenode method)": [[16, "buildstream.node.SequenceNode.mapping_at", false]], "mappingnode (class in buildstream.node)": [[16, "buildstream.node.MappingNode", false]], "mark_directory() (sandbox method)": [[19, "buildstream.sandbox.sandbox.Sandbox.mark_directory", false]], "mark_download_url() (source method)": [[21, "buildstream.source.Source.mark_download_url", false]], "mark_download_url() (sourcefetcher method)": [[21, "buildstream.source.SourceFetcher.mark_download_url", false]], "missing_file (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.MISSING_FILE", false]], "missing_project_conf (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.MISSING_PROJECT_CONF", false]], "module": [[11, "module-buildstream", false], [12, "module-buildstream.buildelement", false], [13, "module-buildstream.downloadablefilesource", false], [14, "module-buildstream.element", false], [15, "module-buildstream.exceptions", false], [16, "module-buildstream.node", false], [17, "module-buildstream.plugin", false], [18, "module-buildstream.sandbox", false], [19, "module-buildstream.sandbox.sandbox", false], [20, "module-buildstream.scriptelement", false], [21, "module-buildstream.source", false], [22, "module-buildstream.sourcemirror", false], [23, "module-buildstream.storage", false], [24, "module-buildstream.storage.directory", false], [25, "module-buildstream.types", false], [26, "module-buildstream.utils", false], [33, "module-elements.compose", false], [34, "module-elements.filter", false], [35, "module-elements.import", false], [36, "module-elements.junction", false], [37, "module-elements.link", false], [38, "module-elements.manual", false], [39, "module-elements.script", false], [40, "module-elements.stack", false], [75, "module-sources.local", false], [76, "module-sources.remote", false], [77, "module-sources.tar", false], [78, "module-sources.workspace", false]], "move_atomic() (in module buildstream.utils)": [[26, "buildstream.utils.move_atomic", false]], "mtime (filestat attribute)": [[24, "buildstream.storage.directory.FileStat.mtime", false]], "name (fastenum attribute)": [[25, "buildstream.types.FastEnum.name", false]], "name (plugin attribute)": [[17, "buildstream.plugin.Plugin.name", false]], "node (class in buildstream.node)": [[16, "buildstream.node.Node", false]], "node_at() (sequencenode method)": [[16, "buildstream.node.SequenceNode.node_at", false]], "node_get_project_path() (plugin method)": [[17, "buildstream.plugin.Plugin.node_get_project_path", false]], "node_subst_sequence_vars() (element method)": [[14, "buildstream.element.Element.node_subst_sequence_vars", false]], "node_subst_vars() (element method)": [[14, "buildstream.element.Element.node_subst_vars", false]], "normal_name (element attribute)": [[14, "buildstream.element.Element.normal_name", false]], "open_directory() (directory method)": [[24, "buildstream.storage.directory.Directory.open_directory", false]], "open_file() (directory method)": [[24, "buildstream.storage.directory.Directory.open_file", false]], "overlapaction (class in buildstream.types)": [[25, "buildstream.types.OverlapAction", false]], "overlaps (corewarnings attribute)": [[25, "buildstream.types.CoreWarnings.OVERLAPS", false]], "overwritten (filelistresult attribute)": [[26, "buildstream.utils.FileListResult.overwritten", false]], "path (dependencyconfiguration attribute)": [[14, "buildstream.element.DependencyConfiguration.path", false]], "pipeline (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.PIPELINE", false]], "platform (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.PLATFORM", false]], "plugin": [[68, "term-Plugin", true]], "plugin (class in buildstream.plugin)": [[17, "buildstream.plugin.Plugin", false]], "plugin (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.PLUGIN", false]], "preflight() (buildelement method)": [[12, "buildstream.buildelement.BuildElement.preflight", false]], "preflight() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.preflight", false]], "preflight() (plugin method)": [[17, "buildstream.plugin.Plugin.preflight", false]], "preflight() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.preflight", false]], "profile (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.PROFILE", false]], "prog_not_found (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.PROG_NOT_FOUND", false]], "programnotfounderror": [[26, "buildstream.utils.ProgramNotFoundError", false]], "proj_path_invalid (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.PROJ_PATH_INVALID", false]], "proj_path_invalid_kind (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.PROJ_PATH_INVALID_KIND", false]], "project": [[68, "term-Project", true]], "project_name (element attribute)": [[14, "buildstream.element.Element.project_name", false]], "protected_variable_redefined (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.PROTECTED_VARIABLE_REDEFINED", false]], "provenanceinformation (class in buildstream.node)": [[16, "buildstream.node.ProvenanceInformation", false]], "readlink() (directory method)": [[24, "buildstream.storage.directory.Directory.readlink", false]], "recursive_include (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.RECURSIVE_INCLUDE", false]], "ref_not_in_track (corewarnings attribute)": [[25, "buildstream.types.CoreWarnings.REF_NOT_IN_TRACK", false]], "regular_file (filetype attribute)": [[24, "buildstream.storage.directory.FileType.REGULAR_FILE", false]], "remote (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.REMOTE", false]], "remote cache": [[68, "term-Remote-Cache", true]], "remove() (directory method)": [[24, "buildstream.storage.directory.Directory.remove", false]], "rename() (directory method)": [[24, "buildstream.storage.directory.Directory.rename", false]], "run() (sandbox method)": [[19, "buildstream.sandbox.sandbox.Sandbox.run", false]], "run_cleanup_commands() (element method)": [[14, "buildstream.element.Element.run_cleanup_commands", false]], "safe_copy() (in module buildstream.utils)": [[26, "buildstream.utils.safe_copy", false]], "safe_del() (mappingnode method)": [[16, "buildstream.node.MappingNode.safe_del", false]], "safe_link() (in module buildstream.utils)": [[26, "buildstream.utils.safe_link", false]], "safe_remove() (in module buildstream.utils)": [[26, "buildstream.utils.safe_remove", false]], "sandbox (class in buildstream.sandbox.sandbox)": [[19, "buildstream.sandbox.sandbox.Sandbox", false]], "sandbox (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.SANDBOX", false]], "sandboxcommanderror": [[19, "buildstream.sandbox.sandbox.SandboxCommandError", false]], "save_file_atomic() (in module buildstream.utils)": [[26, "buildstream.utils.save_file_atomic", false]], "scalar_at() (sequencenode method)": [[16, "buildstream.node.SequenceNode.scalar_at", false]], "scalarnode (class in buildstream.node)": [[16, "buildstream.node.ScalarNode", false]], "scriptelement (class in buildstream.scriptelement)": [[20, "buildstream.scriptelement.ScriptElement", false]], "search() (element method)": [[14, "buildstream.element.Element.search", false]], "sequence_at() (sequencenode method)": [[16, "buildstream.node.SequenceNode.sequence_at", false]], "sequencenode (class in buildstream.node)": [[16, "buildstream.node.SequenceNode", false]], "set_environment() (sandbox method)": [[19, "buildstream.sandbox.sandbox.Sandbox.set_environment", false]], "set_install_root() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.set_install_root", false]], "set_public_data() (element method)": [[14, "buildstream.element.Element.set_public_data", false]], "set_ref() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.set_ref", false]], "set_ref() (source method)": [[21, "buildstream.source.Source.set_ref", false]], "set_root_read_only() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.set_root_read_only", false]], "set_supported_aliases() (sourcemirror method)": [[22, "buildstream.sourcemirror.SourceMirror.set_supported_aliases", false]], "set_work_dir() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.set_work_dir", false]], "set_work_directory() (sandbox method)": [[19, "buildstream.sandbox.sandbox.Sandbox.set_work_directory", false]], "setup() (in module buildstream.scriptelement)": [[20, "buildstream.scriptelement.setup", false]], "sha256sum() (in module buildstream.utils)": [[26, "buildstream.utils.sha256sum", false]], "size (filestat attribute)": [[24, "buildstream.storage.directory.FileStat.size", false]], "source": [[68, "term-Source", true]], "source (class in buildstream.source)": [[21, "buildstream.source.Source", false]], "source (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.SOURCE", false]], "sourceerror": [[21, "buildstream.source.SourceError", false]], "sourcefetcher (class in buildstream.source)": [[21, "buildstream.source.SourceFetcher", false]], "sourcemirror (class in buildstream.sourcemirror)": [[22, "buildstream.sourcemirror.SourceMirror", false]], "sourcemirrorerror": [[22, "buildstream.sourcemirror.SourceMirrorError", false]], "sourceref (in module buildstream.types)": [[25, "buildstream.types.SourceRef", false]], "sources() (element method)": [[14, "buildstream.element.Element.sources", false]], "sources.local": [[75, "module-sources.local", false]], "sources.remote": [[76, "module-sources.remote", false]], "sources.tar": [[77, "module-sources.tar", false]], "sources.workspace": [[78, "module-sources.workspace", false]], "stage() (buildelement method)": [[12, "buildstream.buildelement.BuildElement.stage", false]], "stage() (element method)": [[14, "buildstream.element.Element.stage", false]], "stage() (scriptelement method)": [[20, "buildstream.scriptelement.ScriptElement.stage", false]], "stage() (source method)": [[21, "buildstream.source.Source.stage", false]], "stage_artifact() (element method)": [[14, "buildstream.element.Element.stage_artifact", false]], "stage_dependency_artifacts() (element method)": [[14, "buildstream.element.Element.stage_dependency_artifacts", false]], "stage_directory() (source method)": [[21, "buildstream.source.Source.stage_directory", false]], "stage_sources() (element method)": [[14, "buildstream.element.Element.stage_sources", false]], "stat() (directory method)": [[24, "buildstream.storage.directory.Directory.stat", false]], "status() (plugin method)": [[17, "buildstream.plugin.Plugin.status", false]], "stream (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.STREAM", false]], "strip_node_info() (mappingnode method)": [[16, "buildstream.node.MappingNode.strip_node_info", false]], "strip_node_info() (node method)": [[16, "buildstream.node.Node.strip_node_info", false]], "strip_node_info() (scalarnode method)": [[16, "buildstream.node.ScalarNode.strip_node_info", false]], "strip_node_info() (sequencenode method)": [[16, "buildstream.node.SequenceNode.strip_node_info", false]], "subproject": [[68, "term-Subproject", true]], "subproject_inconsistent (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.SUBPROJECT_INCONSISTENT", false]], "symlink (filetype attribute)": [[24, "buildstream.storage.directory.FileType.SYMLINK", false]], "target": [[84, "cmdoption-bst-artifact-checkout-arg-TARGET", false], [84, "cmdoption-bst-shell-arg-TARGET", false]], "target_directory": [[84, "cmdoption-bst-init-arg-TARGET_DIRECTORY", false]], "tempdir() (source method)": [[21, "buildstream.source.Source.tempdir", false]], "timed_activity() (plugin method)": [[17, "buildstream.plugin.Plugin.timed_activity", false]], "track() (downloadablefilesource method)": [[13, "buildstream.downloadablefilesource.DownloadableFileSource.track", false]], "track() (source method)": [[21, "buildstream.source.Source.track", false]], "trailing_list_directive (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.TRAILING_LIST_DIRECTIVE", false]], "translate_url() (source method)": [[21, "buildstream.source.Source.translate_url", false]], "translate_url() (sourcemirror method)": [[22, "buildstream.sourcemirror.SourceMirror.translate_url", false]], "unaliased_url (corewarnings attribute)": [[25, "buildstream.types.CoreWarnings.UNALIASED_URL", false]], "unresolved_variable (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.UNRESOLVED_VARIABLE", false]], "unstaged_files (corewarnings attribute)": [[25, "buildstream.types.CoreWarnings.UNSTAGED_FILES", false]], "unsupported_plugin (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.UNSUPPORTED_PLUGIN", false]], "unsupported_project (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.UNSUPPORTED_PROJECT", false]], "user_assertion (loaderrorreason attribute)": [[15, "buildstream.exceptions.LoadErrorReason.USER_ASSERTION", false]], "util (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.UTIL", false]], "utilerror": [[26, "buildstream.utils.UtilError", false]], "validate_cache() (source method)": [[21, "buildstream.source.Source.validate_cache", false]], "validate_keys() (mappingnode method)": [[16, "buildstream.node.MappingNode.validate_keys", false]], "value (fastenum attribute)": [[25, "buildstream.types.FastEnum.value", false]], "values() (fastenum class method)": [[25, "buildstream.types.FastEnum.values", false]], "values() (mappingnode method)": [[16, "buildstream.node.MappingNode.values", false]], "virtual_fs (errordomain attribute)": [[15, "buildstream.exceptions.ErrorDomain.VIRTUAL_FS", false]], "warn() (plugin method)": [[17, "buildstream.plugin.Plugin.warn", false]], "warning (overlapaction attribute)": [[25, "buildstream.types.OverlapAction.WARNING", false]], "workspace": [[68, "term-Workspace", true]]}, "objects": {"": [[11, 0, 0, "-", "buildstream"]], "bst": [[84, 7, 1, "cmdoption-bst-builders", "--builders"], [84, 7, 1, "cmdoption-bst-cache-buildtrees", "--cache-buildtrees"], [84, 7, 1, "cmdoption-bst-colors", "--colors"], [84, 7, 1, "cmdoption-bst-c", "--config"], [84, 7, 1, "cmdoption-bst-debug", "--debug"], [84, 7, 1, "cmdoption-bst-default-mirror", "--default-mirror"], [84, 7, 1, "cmdoption-bst-C", "--directory"], [84, 7, 1, "cmdoption-bst-error-lines", "--error-lines"], [84, 7, 1, "cmdoption-bst-fetchers", "--fetchers"], [84, 7, 1, "cmdoption-bst-log-file", "--log-file"], [84, 7, 1, "cmdoption-bst-max-jobs", "--max-jobs"], [84, 7, 1, "cmdoption-bst-message-lines", "--message-lines"], [84, 7, 1, "cmdoption-bst-network-retries", "--network-retries"], [84, 7, 1, "cmdoption-bst-colors", "--no-colors"], [84, 7, 1, "cmdoption-bst-debug", "--no-debug"], [84, 7, 1, "cmdoption-bst-no-interactive", "--no-interactive"], [84, 7, 1, "cmdoption-bst-strict", "--no-strict"], [84, 7, 1, "cmdoption-bst-verbose", "--no-verbose"], [84, 7, 1, "cmdoption-bst-on-error", "--on-error"], [84, 7, 1, "cmdoption-bst-o", "--option"], [84, 7, 1, "cmdoption-bst-pull-buildtrees", "--pull-buildtrees"], [84, 7, 1, "cmdoption-bst-pushers", "--pushers"], [84, 7, 1, "cmdoption-bst-strict", "--strict"], [84, 7, 1, "cmdoption-bst-verbose", "--verbose"], [84, 7, 1, "cmdoption-bst-version", "--version"], [84, 7, 1, "cmdoption-bst-C", "-C"], [84, 7, 1, "cmdoption-bst-c", "-c"], [84, 7, 1, "cmdoption-bst-o", "-o"]], "bst-artifact-checkout": [[84, 7, 1, "cmdoption-bst-artifact-checkout-artifact-remote", "--artifact-remote"], [84, 7, 1, "cmdoption-bst-artifact-checkout-compression", "--compression"], [84, 7, 1, "cmdoption-bst-artifact-checkout-d", "--deps"], [84, 7, 1, "cmdoption-bst-artifact-checkout-directory", "--directory"], [84, 7, 1, "cmdoption-bst-artifact-checkout-f", "--force"], [84, 7, 1, "cmdoption-bst-artifact-checkout-hardlinks", "--hardlinks"], [84, 7, 1, "cmdoption-bst-artifact-checkout-ignore-project-artifact-remotes", "--ignore-project-artifact-remotes"], [84, 7, 1, "cmdoption-bst-artifact-checkout-integrate", "--integrate"], [84, 7, 1, "cmdoption-bst-artifact-checkout-integrate", "--no-integrate"], [84, 7, 1, "cmdoption-bst-artifact-checkout-tar", "--tar"], [84, 7, 1, "cmdoption-bst-artifact-checkout-d", "-d"], [84, 7, 1, "cmdoption-bst-artifact-checkout-f", "-f"], [84, 7, 1, "cmdoption-bst-artifact-checkout-arg-TARGET", "TARGET"]], "bst-artifact-delete": [[84, 7, 1, "cmdoption-bst-artifact-delete-d", "--deps"], [84, 7, 1, "cmdoption-bst-artifact-delete-d", "-d"], [84, 7, 1, "cmdoption-bst-artifact-delete-arg-ARTIFACTS", "ARTIFACTS"]], "bst-artifact-list-contents": [[84, 7, 1, "cmdoption-bst-artifact-list-contents-l", "--long"], [84, 7, 1, "cmdoption-bst-artifact-list-contents-l", "-l"], [84, 7, 1, "cmdoption-bst-artifact-list-contents-arg-ARTIFACTS", "ARTIFACTS"]], "bst-artifact-log": [[84, 7, 1, "cmdoption-bst-artifact-log-out", "--out"], [84, 7, 1, "cmdoption-bst-artifact-log-arg-ARTIFACTS", "ARTIFACTS"]], "bst-artifact-pull": [[84, 7, 1, "cmdoption-bst-artifact-pull-artifact-remote", "--artifact-remote"], [84, 7, 1, "cmdoption-bst-artifact-pull-d", "--deps"], [84, 7, 1, "cmdoption-bst-artifact-pull-ignore-project-artifact-remotes", "--ignore-project-artifact-remotes"], [84, 7, 1, "cmdoption-bst-artifact-pull-d", "-d"], [84, 7, 1, "cmdoption-bst-artifact-pull-arg-ARTIFACTS", "ARTIFACTS"]], "bst-artifact-push": [[84, 7, 1, "cmdoption-bst-artifact-push-artifact-remote", "--artifact-remote"], [84, 7, 1, "cmdoption-bst-artifact-push-d", "--deps"], [84, 7, 1, "cmdoption-bst-artifact-push-ignore-project-artifact-remotes", "--ignore-project-artifact-remotes"], [84, 7, 1, "cmdoption-bst-artifact-push-d", "-d"], [84, 7, 1, "cmdoption-bst-artifact-push-arg-ARTIFACTS", "ARTIFACTS"]], "bst-artifact-show": [[84, 7, 1, "cmdoption-bst-artifact-show-d", "--deps"], [84, 7, 1, "cmdoption-bst-artifact-show-d", "-d"], [84, 7, 1, "cmdoption-bst-artifact-show-arg-ARTIFACTS", "ARTIFACTS"]], "bst-build": [[84, 7, 1, "cmdoption-bst-build-artifact-remote", "--artifact-remote"], [84, 7, 1, "cmdoption-bst-build-d", "--deps"], [84, 7, 1, "cmdoption-bst-build-ignore-project-artifact-remotes", "--ignore-project-artifact-remotes"], [84, 7, 1, "cmdoption-bst-build-ignore-project-source-remotes", "--ignore-project-source-remotes"], [84, 7, 1, "cmdoption-bst-build-r", "--retry-failed"], [84, 7, 1, "cmdoption-bst-build-source-remote", "--source-remote"], [84, 7, 1, "cmdoption-bst-build-d", "-d"], [84, 7, 1, "cmdoption-bst-build-r", "-r"], [84, 7, 1, "cmdoption-bst-build-arg-ELEMENTS", "ELEMENTS"]], "bst-init": [[84, 7, 1, "cmdoption-bst-init-element-path", "--element-path"], [84, 7, 1, "cmdoption-bst-init-f", "--force"], [84, 7, 1, "cmdoption-bst-init-min-version", "--min-version"], [84, 7, 1, "cmdoption-bst-init-project-name", "--project-name"], [84, 7, 1, "cmdoption-bst-init-f", "-f"], [84, 7, 1, "cmdoption-bst-init-arg-TARGET_DIRECTORY", "TARGET_DIRECTORY"]], "bst-shell": [[84, 7, 1, "cmdoption-bst-shell-artifact-remote", "--artifact-remote"], [84, 7, 1, "cmdoption-bst-shell-b", "--build"], [84, 7, 1, "cmdoption-bst-shell-ignore-project-artifact-remotes", "--ignore-project-artifact-remotes"], [84, 7, 1, "cmdoption-bst-shell-ignore-project-source-remotes", "--ignore-project-source-remotes"], [84, 7, 1, "cmdoption-bst-shell-isolate", "--isolate"], [84, 7, 1, "cmdoption-bst-shell-mount", "--mount"], [84, 7, 1, "cmdoption-bst-shell-source-remote", "--source-remote"], [84, 7, 1, "cmdoption-bst-shell-t", "--use-buildtree"], [84, 7, 1, "cmdoption-bst-shell-b", "-b"], [84, 7, 1, "cmdoption-bst-shell-t", "-t"], [84, 7, 1, "cmdoption-bst-shell-arg-COMMAND", "COMMAND"], [84, 7, 1, "cmdoption-bst-shell-arg-TARGET", "TARGET"]], "bst-show": [[84, 7, 1, "cmdoption-bst-show-d", "--deps"], [84, 7, 1, "cmdoption-bst-show-except", "--except"], [84, 7, 1, "cmdoption-bst-show-f", "--format"], [84, 7, 1, "cmdoption-bst-show-order", "--order"], [84, 7, 1, "cmdoption-bst-show-d", "-d"], [84, 7, 1, "cmdoption-bst-show-f", "-f"], [84, 7, 1, "cmdoption-bst-show-arg-ELEMENTS", "ELEMENTS"]], "bst-source-checkout": [[84, 7, 1, "cmdoption-bst-source-checkout-compression", "--compression"], [84, 7, 1, "cmdoption-bst-source-checkout-d", "--deps"], [84, 7, 1, "cmdoption-bst-source-checkout-directory", "--directory"], [84, 7, 1, "cmdoption-bst-source-checkout-except", "--except"], [84, 7, 1, "cmdoption-bst-source-checkout-f", "--force"], [84, 7, 1, "cmdoption-bst-source-checkout-ignore-project-source-remotes", "--ignore-project-source-remotes"], [84, 7, 1, "cmdoption-bst-source-checkout-include-build-scripts", "--include-build-scripts"], [84, 7, 1, "cmdoption-bst-source-checkout-source-remote", "--source-remote"], [84, 7, 1, "cmdoption-bst-source-checkout-tar", "--tar"], [84, 7, 1, "cmdoption-bst-source-checkout-d", "-d"], [84, 7, 1, "cmdoption-bst-source-checkout-f", "-f"], [84, 7, 1, "cmdoption-bst-source-checkout-arg-ELEMENT", "ELEMENT"]], "bst-source-fetch": [[84, 7, 1, "cmdoption-bst-source-fetch-d", "--deps"], [84, 7, 1, "cmdoption-bst-source-fetch-except", "--except"], [84, 7, 1, "cmdoption-bst-source-fetch-ignore-project-source-remotes", "--ignore-project-source-remotes"], [84, 7, 1, "cmdoption-bst-source-fetch-source-remote", "--source-remote"], [84, 7, 1, "cmdoption-bst-source-fetch-d", "-d"], [84, 7, 1, "cmdoption-bst-source-fetch-arg-ELEMENTS", "ELEMENTS"]], "bst-source-push": [[84, 7, 1, "cmdoption-bst-source-push-d", "--deps"], [84, 7, 1, "cmdoption-bst-source-push-except", "--except"], [84, 7, 1, "cmdoption-bst-source-push-ignore-project-source-remotes", "--ignore-project-source-remotes"], [84, 7, 1, "cmdoption-bst-source-push-source-remote", "--source-remote"], [84, 7, 1, "cmdoption-bst-source-push-d", "-d"], [84, 7, 1, "cmdoption-bst-source-push-arg-ELEMENTS", "ELEMENTS"]], "bst-source-track": [[84, 7, 1, "cmdoption-bst-source-track-J", "--cross-junctions"], [84, 7, 1, "cmdoption-bst-source-track-d", "--deps"], [84, 7, 1, "cmdoption-bst-source-track-except", "--except"], [84, 7, 1, "cmdoption-bst-source-track-J", "-J"], [84, 7, 1, "cmdoption-bst-source-track-d", "-d"], [84, 7, 1, "cmdoption-bst-source-track-arg-ELEMENTS", "ELEMENTS"]], "bst-workspace-close": [[84, 7, 1, "cmdoption-bst-workspace-close-a", "--all"], [84, 7, 1, "cmdoption-bst-workspace-close-remove-dir", "--remove-dir"], [84, 7, 1, "cmdoption-bst-workspace-close-a", "-a"], [84, 7, 1, "cmdoption-bst-workspace-close-arg-ELEMENTS", "ELEMENTS"]], "bst-workspace-open": [[84, 7, 1, "cmdoption-bst-workspace-open-directory", "--directory"], [84, 7, 1, "cmdoption-bst-workspace-open-f", "--force"], [84, 7, 1, "cmdoption-bst-workspace-open-ignore-project-source-remotes", "--ignore-project-source-remotes"], [84, 7, 1, "cmdoption-bst-workspace-open-no-checkout", "--no-checkout"], [84, 7, 1, "cmdoption-bst-workspace-open-source-remote", "--source-remote"], [84, 7, 1, "cmdoption-bst-workspace-open-f", "-f"], [84, 7, 1, "cmdoption-bst-workspace-open-arg-ELEMENTS", "ELEMENTS"]], "bst-workspace-reset": [[84, 7, 1, "cmdoption-bst-workspace-reset-a", "--all"], [84, 7, 1, "cmdoption-bst-workspace-reset-soft", "--soft"], [84, 7, 1, "cmdoption-bst-workspace-reset-a", "-a"], [84, 7, 1, "cmdoption-bst-workspace-reset-arg-ELEMENTS", "ELEMENTS"]], "buildstream": [[12, 0, 0, "-", "buildelement"], [13, 0, 0, "-", "downloadablefilesource"], [14, 0, 0, "-", "element"], [15, 0, 0, "-", "exceptions"], [16, 0, 0, "-", "node"], [17, 0, 0, "-", "plugin"], [18, 0, 0, "-", "sandbox"], [20, 0, 0, "-", "scriptelement"], [21, 0, 0, "-", "source"], [22, 0, 0, "-", "sourcemirror"], [23, 0, 0, "-", "storage"], [25, 0, 0, "-", "types"], [26, 0, 0, "-", "utils"]], "buildstream.buildelement": [[12, 1, 1, "", "BuildElement"]], "buildstream.buildelement.BuildElement": [[12, 2, 1, "", "assemble"], [12, 2, 1, "", "configure"], [12, 2, 1, "", "configure_dependencies"], [12, 2, 1, "", "configure_sandbox"], [12, 2, 1, "", "generate_script"], [12, 2, 1, "", "get_unique_key"], [12, 2, 1, "", "preflight"], [12, 2, 1, "", "stage"]], "buildstream.downloadablefilesource": [[13, 1, 1, "", "DownloadableFileSource"]], "buildstream.downloadablefilesource.DownloadableFileSource": [[13, 3, 1, "", "COMMON_CONFIG_KEYS"], [13, 2, 1, "", "configure"], [13, 2, 1, "", "fetch"], [13, 2, 1, "", "get_ref"], [13, 2, 1, "", "get_unique_key"], [13, 2, 1, "", "is_cached"], [13, 2, 1, "", "load_ref"], [13, 2, 1, "", "preflight"], [13, 2, 1, "", "set_ref"], [13, 2, 1, "", "track"]], "buildstream.element": [[14, 1, 1, "", "DependencyConfiguration"], [14, 1, 1, "", "Element"], [14, 4, 1, "", "ElementError"]], "buildstream.element.DependencyConfiguration": [[14, 3, 1, "", "config"], [14, 3, 1, "", "element"], [14, 3, 1, "", "path"]], "buildstream.element.Element": [[14, 3, 1, "", "BST_ARTIFACT_VERSION"], [14, 3, 1, "", "BST_ELEMENT_HAS_ARTIFACT"], [14, 3, 1, "", "BST_FORBID_BDEPENDS"], [14, 3, 1, "", "BST_FORBID_RDEPENDS"], [14, 3, 1, "", "BST_FORBID_SOURCES"], [14, 3, 1, "", "BST_RUN_COMMANDS"], [14, 3, 1, "", "BST_STRICT_REBUILD"], [14, 2, 1, "", "assemble"], [14, 2, 1, "", "compute_manifest"], [14, 2, 1, "", "configure_dependencies"], [14, 2, 1, "", "configure_sandbox"], [14, 2, 1, "", "dependencies"], [14, 2, 1, "", "generate_script"], [14, 2, 1, "", "get_artifact_name"], [14, 2, 1, "", "get_environment"], [14, 2, 1, "", "get_public_data"], [14, 2, 1, "", "get_variable"], [14, 2, 1, "", "integrate"], [14, 2, 1, "", "node_subst_sequence_vars"], [14, 2, 1, "", "node_subst_vars"], [14, 3, 1, "", "normal_name"], [14, 3, 1, "", "project_name"], [14, 2, 1, "", "run_cleanup_commands"], [14, 2, 1, "", "search"], [14, 2, 1, "", "set_public_data"], [14, 2, 1, "", "sources"], [14, 2, 1, "", "stage"], [14, 2, 1, "", "stage_artifact"], [14, 2, 1, "", "stage_dependency_artifacts"], [14, 2, 1, "", "stage_sources"]], "buildstream.exceptions": [[15, 1, 1, "", "ErrorDomain"], [15, 1, 1, "", "LoadErrorReason"]], "buildstream.exceptions.ErrorDomain": [[15, 3, 1, "", "APP"], [15, 3, 1, "", "ARTIFACT"], [15, 3, 1, "", "CAS"], [15, 3, 1, "", "ELEMENT"], [15, 3, 1, "", "IMPL"], [15, 3, 1, "", "LOAD"], [15, 3, 1, "", "PIPELINE"], [15, 3, 1, "", "PLATFORM"], [15, 3, 1, "", "PLUGIN"], [15, 3, 1, "", "PROFILE"], [15, 3, 1, "", "PROG_NOT_FOUND"], [15, 3, 1, "", "REMOTE"], [15, 3, 1, "", "SANDBOX"], [15, 3, 1, "", "SOURCE"], [15, 3, 1, "", "STREAM"], [15, 3, 1, "", "UTIL"], [15, 3, 1, "", "VIRTUAL_FS"]], "buildstream.exceptions.LoadErrorReason": [[15, 3, 1, "", "BAD_CHARACTERS_IN_NAME"], [15, 3, 1, "", "BAD_ELEMENT_SUFFIX"], [15, 3, 1, "", "CIRCULAR_DEPENDENCY"], [15, 3, 1, "", "CIRCULAR_REFERENCE"], [15, 3, 1, "", "CIRCULAR_REFERENCE_VARIABLE"], [15, 3, 1, "", "CONFLICTING_JUNCTION"], [15, 3, 1, "", "EXPRESSION_FAILED"], [15, 3, 1, "", "ILLEGAL_COMPOSITE"], [15, 3, 1, "", "INVALID_DATA"], [15, 3, 1, "", "INVALID_DEPENDENCY_CONFIG"], [15, 3, 1, "", "INVALID_JUNCTION"], [15, 3, 1, "", "INVALID_SYMBOL_NAME"], [15, 3, 1, "", "INVALID_YAML"], [15, 3, 1, "", "LINK_FORBIDDEN_DEPENDENCIES"], [15, 3, 1, "", "LOADING_DIRECTORY"], [15, 3, 1, "", "MISSING_FILE"], [15, 3, 1, "", "MISSING_PROJECT_CONF"], [15, 3, 1, "", "PROJ_PATH_INVALID"], [15, 3, 1, "", "PROJ_PATH_INVALID_KIND"], [15, 3, 1, "", "PROTECTED_VARIABLE_REDEFINED"], [15, 3, 1, "", "RECURSIVE_INCLUDE"], [15, 3, 1, "", "SUBPROJECT_INCONSISTENT"], [15, 3, 1, "", "TRAILING_LIST_DIRECTIVE"], [15, 3, 1, "", "UNRESOLVED_VARIABLE"], [15, 3, 1, "", "UNSUPPORTED_PLUGIN"], [15, 3, 1, "", "UNSUPPORTED_PROJECT"], [15, 3, 1, "", "USER_ASSERTION"]], "buildstream.node": [[16, 1, 1, "", "MappingNode"], [16, 1, 1, "", "Node"], [16, 1, 1, "", "ProvenanceInformation"], [16, 1, 1, "", "ScalarNode"], [16, 1, 1, "", "SequenceNode"]], "buildstream.node.MappingNode": [[16, 2, 1, "", "clone"], [16, 2, 1, "", "get_bool"], [16, 2, 1, "", "get_enum"], [16, 2, 1, "", "get_int"], [16, 2, 1, "", "get_mapping"], [16, 2, 1, "", "get_node"], [16, 2, 1, "", "get_scalar"], [16, 2, 1, "", "get_sequence"], [16, 2, 1, "", "get_str"], [16, 2, 1, "", "get_str_list"], [16, 2, 1, "", "items"], [16, 2, 1, "", "keys"], [16, 2, 1, "", "safe_del"], [16, 2, 1, "", "strip_node_info"], [16, 2, 1, "", "validate_keys"], [16, 2, 1, "", "values"]], "buildstream.node.Node": [[16, 2, 1, "", "clone"], [16, 2, 1, "", "from_dict"], [16, 2, 1, "", "get_provenance"], [16, 2, 1, "", "strip_node_info"]], "buildstream.node.ScalarNode": [[16, 2, 1, "", "as_bool"], [16, 2, 1, "", "as_enum"], [16, 2, 1, "", "as_int"], [16, 2, 1, "", "as_str"], [16, 2, 1, "", "clone"], [16, 2, 1, "", "is_none"], [16, 2, 1, "", "strip_node_info"]], "buildstream.node.SequenceNode": [[16, 2, 1, "", "append"], [16, 2, 1, "", "as_str_list"], [16, 2, 1, "", "clone"], [16, 2, 1, "", "mapping_at"], [16, 2, 1, "", "node_at"], [16, 2, 1, "", "scalar_at"], [16, 2, 1, "", "sequence_at"], [16, 2, 1, "", "strip_node_info"]], "buildstream.plugin": [[17, 1, 1, "", "Plugin"]], "buildstream.plugin.Plugin": [[17, 3, 1, "", "BST_MIN_VERSION"], [17, 3, 1, "", "BST_PLUGIN_DEPRECATED"], [17, 3, 1, "", "BST_PLUGIN_DEPRECATION_MESSAGE"], [17, 2, 1, "", "blocking_activity"], [17, 2, 1, "", "call"], [17, 2, 1, "", "check_output"], [17, 2, 1, "", "configure"], [17, 2, 1, "", "debug"], [17, 2, 1, "", "get_kind"], [17, 2, 1, "", "get_unique_key"], [17, 2, 1, "", "info"], [17, 2, 1, "", "log"], [17, 3, 1, "", "name"], [17, 2, 1, "", "node_get_project_path"], [17, 2, 1, "", "preflight"], [17, 2, 1, "", "status"], [17, 2, 1, "", "timed_activity"], [17, 2, 1, "", "warn"]], "buildstream.sandbox": [[19, 0, 0, "-", "sandbox"]], "buildstream.sandbox.sandbox": [[19, 1, 1, "", "Sandbox"], [19, 4, 1, "", "SandboxCommandError"]], "buildstream.sandbox.sandbox.Sandbox": [[19, 2, 1, "", "batch"], [19, 2, 1, "", "get_virtual_directory"], [19, 2, 1, "", "mark_directory"], [19, 2, 1, "", "run"], [19, 2, 1, "", "set_environment"], [19, 2, 1, "", "set_work_directory"]], "buildstream.scriptelement": [[20, 1, 1, "", "ScriptElement"], [20, 5, 1, "", "setup"]], "buildstream.scriptelement.ScriptElement": [[20, 3, 1, "", "BST_FORBID_RDEPENDS"], [20, 3, 1, "", "BST_FORBID_SOURCES"], [20, 3, 1, "", "BST_STRICT_REBUILD"], [20, 2, 1, "", "add_commands"], [20, 2, 1, "", "assemble"], [20, 2, 1, "", "configure_sandbox"], [20, 2, 1, "", "get_unique_key"], [20, 2, 1, "", "layout_add"], [20, 2, 1, "", "preflight"], [20, 2, 1, "", "set_install_root"], [20, 2, 1, "", "set_root_read_only"], [20, 2, 1, "", "set_work_dir"], [20, 2, 1, "", "stage"]], "buildstream.source": [[21, 1, 1, "", "AliasSubstitution"], [21, 1, 1, "", "Source"], [21, 4, 1, "", "SourceError"], [21, 1, 1, "", "SourceFetcher"]], "buildstream.source.Source": [[21, 3, 1, "", "BST_REQUIRES_PREVIOUS_SOURCES_FETCH"], [21, 3, 1, "", "BST_REQUIRES_PREVIOUS_SOURCES_STAGE"], [21, 3, 1, "", "BST_REQUIRES_PREVIOUS_SOURCES_TRACK"], [21, 3, 1, "", "BST_STAGE_VIRTUAL_DIRECTORY"], [21, 3, 1, "", "COMMON_CONFIG_KEYS"], [21, 2, 1, "", "fetch"], [21, 2, 1, "", "get_mirror_directory"], [21, 2, 1, "", "get_project_directory"], [21, 2, 1, "", "get_ref"], [21, 2, 1, "", "get_source_fetchers"], [21, 2, 1, "", "init_workspace"], [21, 2, 1, "", "init_workspace_directory"], [21, 2, 1, "", "is_cached"], [21, 2, 1, "", "is_resolved"], [21, 2, 1, "", "load_ref"], [21, 2, 1, "", "mark_download_url"], [21, 2, 1, "", "set_ref"], [21, 2, 1, "", "stage"], [21, 2, 1, "", "stage_directory"], [21, 2, 1, "", "tempdir"], [21, 2, 1, "", "track"], [21, 2, 1, "", "translate_url"], [21, 2, 1, "", "validate_cache"]], "buildstream.source.SourceFetcher": [[21, 2, 1, "", "fetch"], [21, 2, 1, "", "mark_download_url"]], "buildstream.sourcemirror": [[22, 1, 1, "", "SourceMirror"], [22, 4, 1, "", "SourceMirrorError"]], "buildstream.sourcemirror.SourceMirror": [[22, 3, 1, "", "BST_MIN_VERSION"], [22, 2, 1, "", "set_supported_aliases"], [22, 2, 1, "", "translate_url"]], "buildstream.storage": [[24, 0, 0, "-", "directory"]], "buildstream.storage.directory": [[24, 1, 1, "", "Directory"], [24, 4, 1, "", "DirectoryError"], [24, 1, 1, "", "FileStat"], [24, 1, 1, "", "FileType"]], "buildstream.storage.directory.Directory": [[24, 2, 1, "", "exists"], [24, 2, 1, "", "export_to_tar"], [24, 2, 1, "", "file_digest"], [24, 2, 1, "", "import_files"], [24, 2, 1, "", "import_single_file"], [24, 2, 1, "", "isdir"], [24, 2, 1, "", "isfile"], [24, 2, 1, "", "islink"], [24, 2, 1, "", "list_relative_paths"], [24, 2, 1, "", "open_directory"], [24, 2, 1, "", "open_file"], [24, 2, 1, "", "readlink"], [24, 2, 1, "", "remove"], [24, 2, 1, "", "rename"], [24, 2, 1, "", "stat"]], "buildstream.storage.directory.FileStat": [[24, 3, 1, "", "executable"], [24, 3, 1, "", "file_type"], [24, 3, 1, "", "mtime"], [24, 3, 1, "", "size"]], "buildstream.storage.directory.FileType": [[24, 3, 1, "", "DIRECTORY"], [24, 3, 1, "", "REGULAR_FILE"], [24, 3, 1, "", "SYMLINK"]], "buildstream.types": [[25, 1, 1, "", "CoreWarnings"], [25, 1, 1, "", "FastEnum"], [25, 1, 1, "", "OverlapAction"], [25, 6, 1, "", "SourceRef"]], "buildstream.types.CoreWarnings": [[25, 3, 1, "", "OVERLAPS"], [25, 3, 1, "", "REF_NOT_IN_TRACK"], [25, 3, 1, "", "UNALIASED_URL"], [25, 3, 1, "", "UNSTAGED_FILES"]], "buildstream.types.FastEnum": [[25, 3, 1, "", "name"], [25, 3, 1, "", "value"], [25, 2, 1, "", "values"]], "buildstream.types.OverlapAction": [[25, 3, 1, "", "ERROR"], [25, 3, 1, "", "IGNORE"], [25, 3, 1, "", "WARNING"]], "buildstream.utils": [[26, 4, 1, "", "DirectoryExistsError"], [26, 1, 1, "", "FileListResult"], [26, 4, 1, "", "ProgramNotFoundError"], [26, 4, 1, "", "UtilError"], [26, 5, 1, "", "copy_files"], [26, 5, 1, "", "get_bst_version"], [26, 5, 1, "", "get_host_tool"], [26, 5, 1, "", "get_umask"], [26, 5, 1, "", "glob"], [26, 5, 1, "", "link_files"], [26, 5, 1, "", "list_relative_paths"], [26, 5, 1, "", "move_atomic"], [26, 5, 1, "", "safe_copy"], [26, 5, 1, "", "safe_link"], [26, 5, 1, "", "safe_remove"], [26, 5, 1, "", "save_file_atomic"], [26, 5, 1, "", "sha256sum"]], "buildstream.utils.FileListResult": [[26, 3, 1, "", "failed_attributes"], [26, 3, 1, "", "files_written"], [26, 3, 1, "", "ignored"], [26, 3, 1, "", "overwritten"]], "elements": [[33, 0, 0, "-", "compose"], [34, 0, 0, "-", "filter"], [35, 0, 0, "-", "import"], [36, 0, 0, "-", "junction"], [37, 0, 0, "-", "link"], [38, 0, 0, "-", "manual"], [39, 0, 0, "-", "script"], [40, 0, 0, "-", "stack"]], "sources": [[75, 0, 0, "-", "local"], [76, 0, 0, "-", "remote"], [77, 0, 0, "-", "tar"], [78, 0, 0, "-", "workspace"]]}, "objnames": {"0": ["py", "module", "Python module"], "1": ["py", "class", "Python class"], "2": ["py", "method", "Python method"], "3": ["py", "attribute", "Python attribute"], "4": ["py", "exception", "Python exception"], "5": ["py", "function", "Python function"], "6": ["py", "data", "Python data"], "7": ["std", "cmdoption", "program option"]}, "objtypes": {"0": "py:module", "1": "py:class", "2": "py:method", "3": "py:attribute", "4": "py:exception", "5": "py:function", "6": "py:data", "7": "std:cmdoption"}, "terms": {"": [0, 2, 3, 4, 5, 8, 9, 12, 13, 14, 16, 17, 20, 21, 24, 25, 26, 31, 32, 33, 34, 36, 39, 41, 43, 44, 45, 46, 48, 50, 53, 55, 56, 57, 58, 59, 60, 62, 63, 65, 68, 69, 73, 76, 77, 79, 80, 81, 82, 83, 84, 85, 86], "0": [9, 12, 14, 17, 24, 31, 32, 33, 34, 35, 38, 39, 41, 42, 43, 45, 48, 50, 58, 59, 60, 62, 63, 73, 79, 80, 81, 82, 83, 85], "00": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "01": [31, 60, 62, 79], "0100": 32, "023013": 32, "024279": 31, "025553785": 32, "03": [62, 63, 79], "04": [31, 62, 63], "047870": 63, "053250": 32, "05bb664034f1bf5b24ab99003b7e0edaa4ed9a890d67eb4677cd0edf81fd1d7b": 31, "06": [31, 32, 58, 60, 62, 63, 79, 80, 81, 82, 83], "066774": 79, "06c0be7b": 60, "06c0be7b5c5692db9a37354e4e1210104d4ebd717f35d49e03ff7508ba3e1bb2": 60, "07": [32, 80], "076727": 80, "077568920": 32, "07db7692": 32, "07db7692b3828f7021ae8c1b0b6f715801e64cf24fcb1621a67b47c110fc3c70": 32, "08": [79, 82], "0848bf17": 82, "0848bf1769eb7f774cfe146212b0cfe56f9d7b7320832c93a7823e50ab993138": 82, "09": [58, 63], "0babd9cc": 58, "0babd9cc5beb41704ffe9a110ed8b4ca69ff8241a630b46176052544c07a35b2": 58, "0x7f4731ac6960": 84, "0x7f4731ac69c0": 84, "0x7f4731ac6a20": 84, "1": [8, 14, 15, 16, 17, 26, 31, 45, 50, 52, 54, 55, 56, 58, 59, 60, 62, 63, 69, 70, 72, 73, 79, 80, 81, 82, 83, 85], "10": [15, 31, 32, 58, 60, 62, 63, 79, 80, 81, 82, 83, 85], "100": [56, 85], "1001": [43, 45], "1003": [43, 45], "100m": 85, "101540": 79, "101548": 79, "101554": 58, "101601": 58, "101602": 58, "101603": 58, "101607": 32, "101613": 32, "101618": 80, "101625": 80, "101626": 80, "101643": 81, "101645": 82, "101652": 82, "101653": 82, "101658": 63, "101705": 63, "101709": 63, "101715": 62, "101722": 62, "101726": 62, "101728": 64, "101739": 60, "101741": 83, "101748": 83, "101801": 31, "101804": 31, "11": [15, 31, 32, 45, 54, 58, 60, 62, 63, 79, 80, 81, 82, 83], "11001": [45, 73, 85], "11002": 85, "12": [15, 62, 63, 79, 85], "123": 0, "1234": 13, "127": [41, 42], "128": 17, "13": [15, 58], "130867": 80, "1321009871": [24, 45], "14": [15, 32], "1408": 79, "141202": 59, "14183452": 80, "14183452db56642e12bc721cd3501237d397ac87b1f24988665d69177fe488cb": 80, "142350": 79, "144630": 31, "1483": 58, "15": [15, 58, 62, 79, 87], "16": [15, 32, 62, 63, 79, 80, 81, 82], "1634": 58, "1663": 58, "17": [15, 60, 62, 63, 83], "1730888140": 79, "1730888153": 79, "1730888154": 58, "1730888164": 58, "1730888166": 32, "1730888167": 32, "1730888174": 32, "1730888175": 32, "1730888176": 32, "1730888177": 32, "1730888178": 80, "1730888185": 80, "1730888186": 80, "1730888187": 80, "1730888188": 80, "1730888199": 59, "1730888200": 59, "1730888201": 59, "1730888203": 81, "1730888204": 81, "1730888205": 82, "1730888214": 82, "1730888218": 63, "1730888229": 63, "1730888234": 63, "1730888235": 62, "1730888247": [62, 64], "1730888248": 64, "1730888259": 60, "1730888261": 83, "1730888269": 83, "1730888279": 31, "1730888280": 31, "1730888281": 31, "1730888283": 31, "1730888284": 31, "1730888285": 31, "1737": 32, "1765": 32, "1793": 32, "179757": 32, "179921": 81, "18": [15, 31, 80], "185127": 60, "1853": 32, "185803": 62, "1881": 32, "19": [15, 42], "1925": 32, "1953": 32, "1981": 32, "1f56d7f344afc5f3f3738600e43bd523ed68d3208419e3d49d1325ff0b83a843": 60, "1hz": 85, "2": [13, 15, 17, 21, 22, 26, 31, 32, 33, 34, 35, 38, 39, 45, 48, 58, 59, 60, 62, 63, 69, 70, 72, 73, 79, 80, 81, 82, 83, 84, 85, 87], "20": [15, 84, 85], "2009": 32, "2011": 45, "2018": 32, "20200816t115912z": 86, "20200903t103837z": 86, "2024": [31, 32, 58, 60, 62, 63, 79, 80, 81, 82, 83], "20241106": [31, 32, 58, 60, 62, 63, 64, 79, 80, 81, 82, 83], "2037": 32, "2096": 80, "21": 15, "2130295a": 63, "2130295a58bedd83aaf3998edb121ccd5942cd749f14b87c19a4be330c90d54f": 63, "2157": 80, "22": [15, 58], "2213": 80, "224801": 31, "2269": 80, "22843536": 80, "2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977": 80, "23": [15, 32, 62, 63, 79], "230": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "2313": 80, "234063": 64, "2357": 80, "24": 15, "25": [15, 32, 80], "250": 85, "250k": 85, "26": [15, 31, 32, 58, 60, 62, 63, 79, 80, 81, 82, 83], "262": 9, "262913": 83, "2644": 59, "2672": 59, "27": 15, "2700": 59, "2765": 59, "27d2a1506044342dd91b6cfd0ca7ca98b5ed14d9c500b8e8beb9b2202dfb21db": 31, "28": [15, 58], "2831": 81, "2861": 81, "287": [58, 60], "2890": 81, "2950": 82, "297": 79, "2eeebcc5": 32, "2eeebcc564b86377e5f029c59454441c88e6ce3d48cdb88e358ea249701e8989": 32, "2f643c2d": 58, "2f643c2da05dd1f300aa1b5aa016a925a6fecfdb3c9ff947c66a97002496aac2": 58, "2to3": 58, "3": [15, 31, 32, 45, 52, 54, 56, 58, 60, 62, 63, 69, 79, 80, 81, 82, 83, 84, 86, 87], "30": 87, "3000": [41, 42], "3081": 82, "311": 54, "3181": 63, "32": [32, 43, 45], "325": 79, "326334": 32, "337185": 63, "340515": 32, "349311": 31, "362886": 80, "38": 9, "39": 60, "396038": 32, "3b986a08": 64, "3cf9edd0": 82, "3cf9edd00c109f063e4a0c30d826420e6d240e59ee10241d59b22a7883ecef4f": 82, "3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639": [58, 82, 83], "3posix": 26, "4": [15, 16, 31, 32, 45, 58, 60, 62, 63, 79, 80, 81, 82, 83, 85, 86], "40": 79, "41": 83, "4195": 63, "42": 58, "4275015f": 80, "4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5": 80, "427621": 31, "43": 81, "44": 58, "440": 45, "448670": 80, "44c6ea15d15476b68d0767c1d410d416f71544e57be572201058f8b3d3b05f83": 73, "45": 82, "461115": 58, "476894": 79, "48": 32, "487641": 82, "49": 32, "49a52215": 80, "49a52215c02358a4bbbb632a74986fa23dc31a36ee8055feffbddc9e3c735d1a": 80, "4aacf94b": 31, "4aacf94bf921018c2224754cdba580590bcdcd26a293bc770f4fd08152369bf4": 31, "4ff941449631ace0d4d203e3483be9dbc9da4540": 46, "5": [15, 31, 32, 43, 58, 60, 85], "50051": [85, 87], "50052": [85, 87], "507356": 83, "5274": 63, "5318": 63, "5363": 62, "54": 58, "553760": 31, "558882": 59, "58": 63, "598029": 32, "5ba078b2324d29d7a455f3a1e754443279bd62fc685c0849bb86ef214cbc51aa": 60, "5ff937aa": 82, "5ff937aaa74f04d961f639c06d636e4a93abbc3a8d289a10ae8084f1d800a2d": 82, "6": [15, 32, 54, 58], "60": 85, "601576": 81, "604786": 80, "617c4e863e3404763b4b4860156e4eb510ea5de1f99384d71465f79ee9495504": 60, "623083": 63, "625510": 31, "64": [43, 45, 69, 87], "644": [31, 60], "644e964a": [62, 79], "644e964abe673c8a8723cf5a3fceeeda44a7bea364f55604fc8132af44a4fb59": [62, 79], "6452": 62, "6497": 64, "6526": 64, "665347": 58, "6749": 60, "677493": 63, "6831": 83, "683298": 83, "6859": 83, "6919": 83, "6947": 83, "6a78c030": [62, 79], "6a78c03097648e558c803c20d8d20f61d3e87e3e616f3a45923704cfe95019af": [62, 79], "6c9f6f68a131ec6381da82f2bff978083ed7f4f7991d931bfa767b7965ebc94b": [76, 77], "6d2e5da0": [62, 64], "6fb0f24a": 58, "6fb0f24a1fee36ba2f486fa0cd0cbedeb8f7a69df15908b9370aaa7799f8353d": 58, "6m": 58, "7": [15, 16, 43, 58, 63], "705887": 82, "7192": 31, "72169f19": 80, "72169f19db218d37d9f9b2a8603c0b75dcd391d3a07e5ef900f95e6fdaaca1ba": 80, "7220": 31, "7249": 31, "7277": 31, "7306": 31, "731437": 59, "7334": 31, "7414": 31, "745122": 32, "7478": 31, "754866": 32, "755": [31, 59, 60, 80, 83], "7553": 31, "75ba510d": 63, "75ba510d9aa73503f87fd9bc44db651a2cf7aca8c67f13d686abbe0b35dced0": 63, "763861": 81, "76f82637": 82, "76f82637c19f9595a18d2e480da027552cb53811e39b7481b2feb767f0d2311d": 82, "788871": 59, "788da21e7c1b5818b7e7b60f7eb75841057ff7e45d362cc223336c606fe47f27": [40, 84], "7981": 86, "7982": 86, "8": [2, 15, 24, 42, 50, 58, 62, 69, 84, 85], "80": 85, "804244": 62, "8088": 84, "80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f": [63, 79], "814005": 64, "816882": 32, "821189": 31, "825d9815": 31, "825d9815607a5d73945c9f1fd457834e02870a54a03bf09619bb754763e24fc8": 31, "826886": 80, "827637": 84, "8276376b077eda104c812e6ec2f488c7c9eea211ce572c83d734c10bf241209f": 84, "829086": 31, "82d7d342": 58, "82d7d3429a94cd01054753bbdc565d3f37069aacba51051764ec0f72129349f7": 58, "894416": 58, "8b3d5352": 81, "8b3d535225335b4842b5d6b98ee6fdffc7340e190f33b79a2043f5c31e43fbd": 81, "8cebf405": 83, "8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07": 83, "8cg84roo": 32, "9": 15, "90136c4": 86, "91": 73, "911b8a6d": 59, "919086": 83, "96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd": 31, "9d4b1147f8cf244b0002ba74bfb0b8dfb3": 43, "A": [2, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 32, 33, 34, 36, 41, 43, 44, 45, 46, 48, 56, 59, 68, 73, 80, 84, 85, 86, 87], "AS": [33, 34, 35, 38, 39, 45, 85], "And": [16, 31, 43, 50, 56, 59, 79, 80, 81, 83], "As": [2, 3, 4, 14, 31, 36, 43, 45, 48, 49, 53, 56, 57, 58, 59, 60, 63, 64, 79, 85, 86], "At": [44, 48, 50, 51, 58, 63, 69], "Be": 84, "But": 68, "By": [31, 35, 36, 43, 45, 55, 65, 84, 85], "For": [0, 2, 9, 10, 12, 13, 14, 16, 17, 20, 21, 22, 31, 34, 36, 39, 41, 43, 44, 45, 48, 50, 52, 54, 55, 56, 57, 61, 62, 63, 68, 69, 73, 82], "If": [0, 1, 2, 8, 9, 12, 13, 14, 16, 17, 19, 20, 21, 24, 25, 26, 32, 34, 36, 41, 42, 43, 45, 46, 48, 50, 52, 53, 54, 55, 56, 59, 63, 65, 68, 69, 76, 80, 84, 85], "In": [0, 1, 3, 4, 5, 9, 12, 16, 17, 21, 24, 26, 31, 32, 33, 34, 36, 40, 43, 44, 45, 46, 47, 48, 50, 52, 53, 54, 55, 56, 58, 59, 60, 63, 64, 68, 72, 73, 79, 80, 81, 82, 83, 84, 85, 86, 87], "It": [0, 2, 9, 12, 13, 14, 16, 17, 21, 24, 25, 26, 31, 34, 36, 39, 40, 41, 43, 45, 48, 52, 54, 55, 56, 58, 63, 68, 73, 76, 77, 83, 85], "No": [9, 31, 48, 84], "Not": [3, 79, 85], "OF": [33, 34, 35, 38, 39, 45, 85], "ON": 69, "OR": [33, 34, 35, 38, 39, 43, 45, 85], "Of": [0, 50], "On": [9, 63, 85, 86, 87], "One": [26, 45, 48, 57, 59, 79, 85], "Or": 45, "Such": [36, 55], "That": 36, "The": [0, 1, 2, 3, 4, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 25, 26, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 43, 44, 45, 47, 48, 49, 50, 54, 55, 56, 57, 58, 59, 60, 62, 63, 67, 68, 69, 72, 74, 77, 78, 79, 82, 83, 84, 85, 86, 87], "There": [2, 3, 9, 16, 36, 43, 45, 48, 50, 52, 54, 58], "These": [0, 4, 9, 35, 45, 48, 52, 53, 56, 58, 61, 68, 69, 82, 83, 84, 85, 87], "To": [0, 2, 9, 10, 31, 32, 40, 41, 42, 43, 45, 49, 53, 55, 56, 65, 68, 69, 72, 77, 79, 80, 81, 82, 83, 84, 85], "Will": [16, 48, 84], "With": [2, 16, 36, 48, 85], "_": [17, 43, 56, 69], "__init__": [17, 48, 57], "__kind": 17, "_agmhsju": 58, "_anchor_nam": 56, "_artifact": 48, "_artifactcach": [0, 48], "_bst": 69, "_build": 12, "_compute_nam": 48, "_context": 48, "_count": 48, "_do_frob": 48, "_effective_alia": 21, "_except": 16, "_exit": 21, "_frontend": 53, "_get_cache_key_for_build": 2, "_get_cache_key_from_artifact": 2, "_get_context": 48, "_internal_link": 56, "_is_main_process": 48, "_messag": 53, "_messeng": 53, "_mirror": 21, "_my_modul": 52, "_pipelineselect": 84, "_profil": 52, "_project": 48, "_proto": 49, "_refpath": 48, "_unlink": 26, "_update_cachekei": 57, "a6a39926": 32, "aarch32": [43, 45], "aarch64": [43, 45], "abbrevi": [84, 85], "abc": 26, "abdc7cb1": 31, "abdc7cb10e1696d9b3c51baf21e83d847f7164f84b3a4627b40bf1ff50322330": 31, "abi": 0, "abil": [50, 65, 87], "abl": [9, 21, 55, 83, 86], "abort": [19, 44, 55, 85], "about": [2, 9, 10, 12, 17, 24, 31, 45, 48, 50, 55, 56, 58, 59, 60, 61, 63, 67, 82, 83, 84, 85, 86], "abov": [1, 4, 9, 16, 34, 43, 45, 48, 50, 54, 55, 56, 60, 69, 73, 79, 80, 81, 82], "absenc": [46, 59], "absens": 25, "absent": 84, "absl": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "absolut": [12, 14, 19, 20, 24, 26, 43, 58, 73, 78, 87], "abstract": [0, 4, 10, 11, 29, 40, 43, 45, 67, 79], "ac": 87, "accept": [25, 45, 48, 72, 84], "access": [11, 13, 14, 16, 25, 36, 40, 43, 45, 48, 50, 52, 73, 81, 83, 85], "accessor": 16, "accident": [45, 85], "accommod": 48, "accompani": [48, 84, 85], "accomplish": [40, 56], "accord": [14, 48, 85], "accordingli": [16, 50, 51, 54], "account": [32, 45, 50, 85], "accru": 45, "acheiv": 72, "achiev": [1, 5, 9, 12, 45, 55, 60, 72], "acloc": [45, 58], "acquir": 16, "across": [3, 36, 43, 44, 45, 48, 55, 63, 83], "act": [4, 5, 36, 59, 62], "action": [0, 14, 25, 48, 50, 85, 87], "activ": [4, 5, 10, 17, 31, 32, 41, 42, 45, 48, 53, 56, 58, 59, 60, 62, 63, 64, 69, 79, 80, 81, 82, 83, 85], "activity_nam": 17, "actual": [2, 14, 21, 31, 43, 45, 46, 48, 49, 50, 80, 85], "ad": [12, 13, 17, 19, 20, 22, 32, 33, 40, 43, 44, 45, 48, 50, 51, 52, 58, 59, 60, 72, 73, 79, 80, 82], "add": [13, 14, 20, 26, 32, 38, 41, 42, 44, 45, 48, 50, 52, 54, 55, 58, 63, 64, 69, 80], "add_command": [11, 20], "addgroup": 58, "addit": [0, 3, 12, 14, 17, 21, 22, 34, 40, 43, 45, 48, 50, 53, 56, 57, 58, 60, 62, 63, 67, 68, 69, 79, 80, 85, 86], "addition": [43, 45, 55], "addr2lin": 58, "address": [0, 8, 21, 48, 66, 69, 73, 85], "addus": 58, "adher": [0, 21, 45, 53], "adjac": [12, 36, 46], "advanc": [0, 10, 44, 55, 56], "advantag": [45, 65], "advert": [48, 50, 56, 63], "advertis": 17, "af6ba39142220687c500f79b4aa2f181d9b24e4": 46, "affect": [2, 4, 9, 12, 13, 17, 20, 31, 43, 45, 48, 50, 57, 59, 63], "after": [0, 2, 4, 8, 12, 13, 14, 17, 20, 31, 43, 44, 47, 48, 50, 54, 55, 64, 68, 79, 82, 83, 85], "again": [0, 10, 14, 21, 32, 45, 55, 59, 82], "against": [0, 14, 17, 31, 36, 55, 63], "aggreg": 85, "agre": [33, 34, 35, 38, 39, 45, 85], "agreement": 48, "ahead": 31, "aim": [2, 48, 55], "aix": 43, "algorithm": [48, 52], "alia": [13, 21, 22, 25, 36, 41, 42, 43, 45, 59, 76, 77, 79, 80, 83, 85], "alias": [21, 22, 36, 41, 42, 59, 63, 73, 79, 80, 83, 85], "alias_overrid": [13, 21], "alias_url": [13, 22], "aliassubstitut": [11, 13, 21, 29, 67], "align": [16, 85], "all": [0, 2, 3, 4, 5, 8, 9, 12, 14, 16, 17, 21, 22, 24, 25, 26, 30, 31, 32, 33, 34, 40, 43, 44, 45, 48, 49, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 63, 64, 65, 72, 73, 79, 80, 81, 82, 83, 84, 85, 87, 89], "all_repo_kind": 57, "alloc": 85, "allow": [0, 2, 3, 4, 8, 9, 10, 13, 14, 15, 16, 17, 19, 21, 22, 24, 36, 37, 38, 39, 40, 43, 44, 45, 46, 47, 48, 52, 53, 55, 59, 60, 62, 63, 64, 65, 68, 73, 79, 80, 82, 83, 84, 85, 87, 91], "allow_non": 16, "allowed_typ": 16, "almost": [2, 26, 31], "alon": 48, "along": [12, 14, 17, 44, 45, 48, 55, 67, 84, 85], "alongsid": 55, "alpha": 84, "alphabet": [43, 84], "alphanumer": [43, 45], "alpin": [31, 32, 43, 59, 60, 62, 63, 80], "alreadi": [0, 1, 2, 8, 10, 21, 24, 26, 43, 44, 45, 48, 55, 56, 57, 63, 69, 73, 83, 84, 85, 86], "alright": 48, "also": [0, 1, 2, 3, 4, 5, 8, 9, 14, 16, 17, 19, 20, 21, 25, 29, 31, 32, 34, 36, 40, 41, 43, 44, 45, 48, 50, 52, 54, 55, 56, 58, 59, 60, 62, 63, 65, 68, 69, 73, 79, 80, 82, 83, 84, 85, 86], "alter": [32, 84], "altern": [0, 12, 13, 17, 21, 22, 30, 32, 45, 56, 63], "alwai": [0, 2, 8, 10, 16, 17, 31, 40, 44, 45, 47, 53, 55, 56, 79, 81, 83, 84, 85], "amend": 0, "amhello": [62, 63, 79], "among": [45, 85], "amount": [17, 56, 85], "ams3": [32, 58, 59, 62, 63, 79, 80, 82, 83], "an": [0, 1, 2, 3, 4, 5, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 30, 31, 32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 50, 52, 55, 57, 58, 59, 60, 62, 63, 68, 69, 72, 76, 77, 79, 80, 81, 82, 83, 84, 85, 86, 88], "analys": 52, "analyz": 52, "anchor": 56, "anchor_nam": 56, "ani": [0, 3, 4, 5, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 31, 33, 34, 35, 36, 38, 39, 40, 43, 44, 45, 46, 47, 48, 50, 51, 55, 56, 57, 58, 59, 61, 63, 64, 65, 69, 73, 79, 80, 82, 83, 84, 85, 86], "annot": [4, 31, 48, 58, 82], "announc": [50, 85], "anoth": [4, 12, 15, 16, 17, 21, 26, 30, 33, 36, 37, 41, 42, 43, 44, 45, 48, 52, 55, 56, 58, 60, 62, 68, 79, 80, 85], "anounc": 50, "ansi": 84, "anyon": [29, 48], "anyth": [4, 14, 16, 43, 51, 53, 56, 58, 63, 82, 83, 85], "anywai": [9, 79], "anywher": 80, "apach": [0, 1, 33, 34, 35, 38, 39, 45, 50, 69, 85], "api": [0, 8, 9, 11, 12, 13, 17, 20, 22, 24, 45, 50, 55, 56, 63, 67, 85, 87], "app": [11, 15, 45, 53], "appear": [0, 1, 13, 21, 44, 48, 54], "append": [11, 16, 21, 58], "appli": [0, 13, 15, 21, 43, 44, 45, 46, 47, 48, 63, 84], "applic": [5, 31, 33, 34, 35, 36, 38, 39, 45, 53, 58, 59, 63, 85], "appos": 17, "approach": [0, 17, 36, 45, 48, 55, 60, 73], "appropi": 12, "appropri": [14, 22, 43, 45, 48, 53, 54, 55, 73], "approv": 50, "approxim": 7, "apt": 55, "ar": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 17, 20, 21, 22, 24, 25, 26, 29, 30, 31, 32, 33, 34, 35, 36, 39, 40, 43, 44, 45, 46, 47, 48, 49, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 63, 64, 68, 69, 72, 73, 79, 80, 81, 82, 83, 84, 85, 86, 87, 89], "arbitari": 9, "arbitrari": [30, 43, 45, 48, 56], "arbitrarili": [43, 63], "arch": [43, 45, 69], "architectur": [8, 43, 48, 61, 63, 65], "archiv": [24, 30, 31, 45, 50, 79], "area": [9, 20, 31, 32, 58, 60, 62, 63, 79, 80, 81, 82, 83], "arg": [17, 48, 79, 84], "argc": [32, 59, 80, 83], "argument": [13, 14, 17, 19, 20, 21, 24, 26, 45, 53, 55, 59, 63, 72, 82, 83, 84], "argv": [32, 59, 80, 83], "aris": [0, 45, 48], "armv7": 65, "armv8": 65, "around": [0, 13, 14, 48, 50, 60], "arp": 58, "arrai": [38, 69], "arriv": 56, "articl": 56, "artifact": [0, 1, 2, 4, 5, 9, 10, 11, 12, 14, 15, 19, 20, 21, 31, 32, 33, 34, 35, 36, 40, 47, 48, 49, 50, 53, 58, 62, 63, 65, 66, 68, 71, 72, 74, 79, 80, 81, 82, 83, 86, 87], "artifact_kei": [12, 20], "artifact_remot": 84, "artifactcach": [0, 48], "artifactdir": 74, "artifacterror": 48, "as_bool": [11, 16], "as_enum": [11, 16], "as_int": [11, 16], "as_str": [11, 16], "as_str_list": [11, 16], "ascii": 43, "asid": [4, 43, 48], "ask": [50, 56, 65, 81, 85], "aspect": [2, 9, 63, 85, 90], "assembl": [4, 9, 11, 14, 20], "assembli": [9, 14, 40], "assert": [15, 17, 48, 55, 56], "asset": [3, 8, 45, 86], "assign": [8, 16, 21, 36, 41, 80, 86], "associ": [17, 21, 73], "assosci": 17, "assum": [0, 1, 21, 31, 41, 43, 45, 82, 84, 85], "asterisk": 43, "atom": [26, 68], "attempt": [10, 15, 24, 26, 48, 69, 84, 85], "attent": 56, "attribut": [9, 14, 21, 24, 25, 26, 31, 43, 45, 48], "audienc": [48, 56, 58], "auditwheel": 54, "augment": 45, "auth": [11, 45, 73, 85], "authent": [84, 86], "author": [13, 44, 45, 48, 50], "auto": [53, 84, 85], "autocomplet": 53, "autoconf": 58, "autogen": [62, 63, 79], "autohead": 58, "autoload": 69, "autom": [50, 65, 79], "autom4t": 58, "automak": [44, 46, 58, 62, 63, 79], "automat": [0, 2, 12, 13, 17, 20, 21, 32, 36, 45, 46, 47, 48, 50, 52, 53, 56, 57, 69, 72, 79, 84, 85], "autoreconf": [58, 62, 63, 79], "autoscan": 58, "autotool": [12, 17, 32, 36, 43, 45, 62, 63, 64, 71, 73, 92], "autoupd": 58, "avail": [1, 2, 9, 10, 13, 21, 42, 43, 45, 48, 52, 53, 68, 69, 73, 84, 85], "avoid": [0, 3, 9, 10, 12, 14, 17, 26, 31, 32, 36, 45, 49, 50, 56, 60, 68, 84, 85, 86], "awai": [40, 45, 48], "awar": [48, 58], "awk": 58, "b": [43, 84], "b4nf6j9j": 60, "b59f4520": 83, "b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af": 83, "b6536670": 32, "b65366701aaab8b2f85afa7158f4036a1f55a8150db42fff45e0830cc8388d73": 32, "b6da6ab": 62, "b6da6abe68420499ec616c50433619027a915e37217b85450a5590a28bd28820": 62, "back": [4, 16, 21, 32, 48, 53, 63, 81, 86], "backbon": 83, "backend": [9, 13, 21, 41, 69], "background": [17, 61], "backslash": 43, "backward": 45, "bad": 48, "bad_characters_in_nam": [11, 15], "bad_element_suffix": [11, 15], "badg": 50, "bahavior": 79, "bail": 17, "balanc": 10, "band": 45, "bandwidth": [10, 85], "bar": [34, 43, 45, 48, 53, 85], "bare": 41, "base": [2, 4, 5, 9, 10, 11, 12, 13, 15, 16, 19, 20, 24, 25, 26, 29, 31, 32, 43, 44, 45, 46, 47, 48, 52, 55, 59, 60, 62, 63, 65, 67, 69, 73, 77, 80, 85, 87], "basemost": 14, "basenam": [58, 76], "baseproject": [12, 43], "baserock": 65, "bash": [45, 84], "bash_complet": 69, "bashrc": 69, "basi": [0, 33, 34, 35, 38, 39, 44, 45, 58, 63, 79, 83, 85], "basic": [6, 10, 28, 38, 44, 48, 56, 61, 81, 82, 83, 85, 86, 87], "batch": [11, 14, 18, 19], "batteri": 57, "baz": 34, "bazel": 86, "bazelbuild": 8, "bb": 86, "bb393e5a": 83, "bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c": 83, "bearer": [13, 85], "becaus": [4, 5, 8, 9, 10, 14, 26, 31, 36, 45, 48, 50, 55, 58, 80, 82, 85, 87], "becom": [0, 9, 56, 57, 73], "been": [0, 2, 8, 10, 12, 13, 14, 15, 17, 20, 21, 22, 31, 43, 45, 50, 55, 56, 58, 59, 62, 72, 73, 74, 79, 82, 84, 85], "beep": 58, "befor": [0, 3, 8, 12, 14, 16, 20, 21, 26, 31, 32, 43, 44, 47, 48, 50, 52, 54, 55, 56, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 84, 85], "beforehand": [16, 20], "begin": [45, 48, 61, 82, 85], "behalf": [10, 12, 13, 21], "behav": [4, 14, 16, 22, 24, 43, 62], "behavior": [0, 20, 22, 24, 28, 31, 44, 45, 55, 59, 72], "behaviour": [21, 43, 53], "behind": [48, 55], "being": [0, 2, 4, 9, 14, 16, 17, 19, 31, 32, 36, 41, 42, 43, 48, 50, 56, 58, 60, 62, 63, 80, 83, 84, 85], "bellow": 32, "belong": [0, 21, 45, 58, 83, 85], "below": [2, 4, 6, 9, 10, 34, 36, 41, 43, 44, 45, 48, 55, 62, 69, 80, 85], "benefici": 85, "benefit": [2, 36], "besid": [48, 55], "best": [56, 85], "better": [0, 17, 36, 43, 45, 48, 52, 82], "between": [2, 3, 5, 10, 43, 45, 48, 52, 63, 80, 85], "beyond": [9, 85], "bfd": 58, "bgd": 87, "bho1gvzz": 31, "big": 48, "bin": [1, 31, 43, 45, 47, 58, 59, 60, 62, 63, 69, 79, 80, 83], "binari": [0, 9, 11, 19, 38, 41, 43, 45, 50, 73, 79, 83], "bind": [9, 45, 86], "bindir": [44, 45, 47, 62, 63, 79], "bisect": 0, "bit": [4, 9, 43, 48], "bitbucket": 65, "black": [48, 55], "blame": 48, "blank": 85, "blkdiscard": 58, "blob": 69, "block": [9, 16, 17, 26, 45, 48, 56, 85], "blocking_act": [11, 17], "blue": 63, "blurb": 50, "bool": [13, 14, 16, 17, 19, 20, 21, 22, 24, 26, 45, 48, 59, 63], "boolean": [16, 34, 63], "bootabl": 65, "bootstrap": [5, 46, 52, 62, 63, 65, 79], "bot": 87, "both": [2, 3, 5, 8, 12, 14, 17, 20, 24, 31, 36, 40, 42, 43, 48, 55, 58, 59, 60, 73, 74, 82, 84, 85], "bottom": 48, "bound": 45, "boundari": [15, 24, 36, 43, 45, 63, 84], "box": 68, "brace": 85, "branch": [13, 21, 36, 50, 84], "brctl": 58, "break": [43, 48, 55, 72, 73, 74], "breif": [21, 22], "breviti": [31, 59], "bridg": 87, "brief": [0, 12, 17, 43, 48, 85], "briefli": 31, "bring": 36, "broken": [44, 48], "brows": [41, 69, 79, 81], "browser": [52, 56], "bst": [1, 4, 5, 8, 9, 12, 13, 14, 15, 16, 17, 21, 30, 32, 34, 36, 37, 39, 40, 41, 42, 43, 44, 45, 46, 47, 52, 56, 57, 60, 62, 64, 65, 68, 69, 76, 77, 85], "bst_artifact_vers": [11, 14], "bst_cython_profil": 52, "bst_cython_trac": 55, "bst_element_has_artifact": [11, 14], "bst_forbid_bdepend": [11, 14], "bst_forbid_rdepend": [11, 14, 20], "bst_forbid_sourc": [11, 14, 20], "bst_force_session_rebuild": 56, "bst_min_vers": [11, 17, 22], "bst_plugin_deprec": [11, 17], "bst_plugin_deprecation_messag": [11, 17], "bst_requires_previous_sources_fetch": [11, 13, 21], "bst_requires_previous_sources_stag": [11, 21], "bst_requires_previous_sources_track": [11, 13, 21], "bst_run_command": [11, 14], "bst_source_cach": 56, "bst_stage_virtual_directori": [11, 21], "bst_strict_rebuild": [11, 14, 20], "bsterror": [14, 21, 22, 24, 26], "bubblewrap": [8, 9, 43, 69, 87], "buffer": [0, 3, 26], "bug": [9, 31, 48, 50, 85], "build": [3, 4, 5, 9, 10, 11, 16, 18, 20, 21, 29, 30, 33, 34, 35, 36, 39, 40, 41, 42, 44, 45, 47, 50, 52, 53, 54, 59, 60, 61, 65, 66, 67, 68, 71, 73, 74, 77, 86, 87], "build_grpc": 49, "buildabl": [31, 50, 60, 63, 72, 80, 81, 84, 85], "buildbarn": 87, "buildbox": [9, 54, 55, 87], "builddir": 74, "buildel": [9, 11, 17, 29, 38, 47, 67, 73, 79], "builder": [84, 85], "buildfarm": 87, "buildgrid": [69, 87], "buildstream": [0, 2, 3, 4, 5, 6, 7, 8, 9, 12, 14, 15, 16, 17, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 36, 40, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 53, 55, 57, 58, 59, 60, 62, 63, 64, 66, 67, 68, 70, 71, 72, 79, 80, 81, 82, 83, 84, 85, 86, 88, 89, 91, 92], "buildstream1": 85, "buildstream2": [74, 85], "buildtim": 39, "buildtrack": 55, "buildtre": [68, 84, 85], "built": [1, 2, 4, 8, 10, 11, 14, 30, 32, 36, 38, 40, 43, 45, 47, 53, 58, 62, 63, 64, 65, 68, 69, 72, 75, 76, 77, 80, 83, 84, 85], "builtin": [28, 43, 67, 79, 82], "bullet": 0, "bump": 14, "bunch": 0, "bunzip2": 58, "burden": 56, "bwrap": 43, "byte": [17, 24, 85], "bz2": 84, "bzcat": 58, "bzip2": 58, "bzr": 73, "c": [1, 32, 43, 45, 54, 58, 60, 65, 69, 82, 84, 87], "c22b89ed": 58, "c22b89ed1a1f72aac4ee28fe220f9efa2c2d4002a1b1c414e659cfb089435960": 58, "c93bc58b": 63, "c93bc58b2fe8afba05ea49c1a941775a15442a56ea826dee5c017641de7c469c": 63, "c_rehash": 58, "ca": [8, 9, 11, 15, 31, 32, 58, 64, 66, 69, 85, 86, 87], "cach": [0, 9, 10, 12, 13, 14, 17, 20, 21, 25, 31, 32, 40, 43, 47, 49, 53, 55, 56, 57, 58, 59, 60, 62, 63, 64, 65, 66, 68, 69, 71, 72, 79, 80, 81, 82, 83, 84, 87], "cache_buildtre": 84, "cache_kei": 48, "cachedir": [74, 85], "cal": 58, "calcul": [2, 3, 26, 45, 65], "call": [1, 11, 12, 13, 14, 17, 19, 20, 21, 24, 25, 26, 31, 32, 52, 55, 56, 58, 59, 60, 62, 63, 64, 68, 69, 79, 80, 81, 82, 83], "call_status_override_on_cancel": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "callabl": [17, 24, 26], "callback": [24, 26], "caller": 48, "callhello": 62, "came": [0, 16], "can": [0, 1, 2, 3, 4, 5, 8, 10, 12, 13, 14, 15, 16, 17, 20, 21, 22, 24, 25, 26, 29, 30, 31, 32, 33, 34, 36, 37, 39, 40, 43, 44, 45, 46, 47, 48, 50, 52, 55, 56, 57, 58, 59, 60, 62, 63, 64, 66, 68, 69, 72, 77, 79, 80, 81, 82, 83, 84, 85, 86, 87, 89], "candid": 55, "cannot": [2, 4, 9, 10, 12, 16, 24, 25, 26, 36, 40, 44, 45, 48, 50, 58, 59, 62, 72, 73, 85], "capabl": [8, 65, 85], "capit": 56, "captur": [2, 3, 14, 24, 56, 58, 59], "care": [0, 12, 13, 14, 17, 20, 45, 48, 50, 52, 85], "carri": [8, 81, 83], "cascach": [0, 55], "casd": [9, 69, 87], "case": [0, 1, 5, 12, 13, 14, 16, 17, 20, 21, 22, 26, 31, 32, 33, 34, 36, 40, 43, 44, 45, 47, 48, 55, 56, 57, 58, 59, 60, 63, 72, 79, 80, 81, 82, 84, 85], "cat": 59, "catch": 21, "categor": 47, "categori": [48, 56], "caus": [8, 9, 17, 25, 31, 32, 36, 45, 48, 60, 63], "caveat": 32, "cc": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "ccmake": 58, "cd": 69, "cdn": [32, 58, 59, 62, 63, 79, 80, 82, 83], "central": [4, 45, 53, 68, 81], "centric": [4, 10], "cert": [45, 73, 84, 85], "certain": [8, 9, 43, 45, 48, 68, 84], "certif": [58, 85], "cfc97a85": 31, "cfc97a856b683a50e0b18ee6073c6283763d6ccc3937717d60f4b7281fa5e562": 31, "cgi": 41, "chain": [15, 20, 82], "chanc": 48, "chang": [0, 2, 9, 12, 14, 17, 20, 21, 31, 36, 40, 43, 45, 48, 54, 55, 56, 57, 58, 59, 60, 64, 65, 72, 73, 74, 79, 81, 83, 84, 85], "channel": [50, 65], "chapter": [31, 56, 58, 59, 60, 63, 79, 80, 82, 83], "char": [31, 32, 59, 80, 83], "charact": [14, 15, 17, 24, 26, 43, 45, 84, 85], "chart": 7, "check": [1, 12, 13, 16, 17, 20, 21, 24, 26, 32, 48, 50, 58, 69, 81], "check_is_dir": 17, "check_is_fil": 17, "check_output": [11, 17], "checklist": [50, 53, 57], "checkout": [8, 14, 32, 40, 45, 50, 55, 81, 85], "checksum": 26, "child": [10, 48], "chipset": 65, "chmod": 1, "choic": 53, "choos": [31, 41, 42, 44, 45, 63], "chosen": 55, "chpasswd": 58, "chroot": [8, 9, 58], "chvt": 58, "ci": [48, 50, 56, 86, 87, 89], "cibuildwheel": 54, "cibw_prerelease_python": 54, "circular": [15, 44], "circular_depend": [11, 15], "circular_refer": [11, 15], "circular_reference_vari": [11, 15], "circumv": 48, "cksum": 58, "class": [0, 4, 10, 11, 15, 19, 24, 25, 26, 29, 30, 43, 47, 52, 53, 67, 79], "classifi": 85, "classmethod": [16, 25], "clean": [0, 50], "cleanli": 53, "cleanup": [0, 14, 21, 85], "clear": [48, 58], "cli": [50, 53, 55], "click": 53, "client": [45, 48, 50, 58, 84, 85, 86, 87], "clone": [11, 16, 41, 42, 69], "close": [26, 34], "closefd": 26, "cluster": [85, 87], "cmake": [12, 58, 65, 69, 73, 85], "cmd": 79, "cmp": 58, "co": [41, 42], "coalesc": 73, "code": [0, 4, 10, 12, 14, 17, 19, 26, 36, 43, 45, 52, 56, 57, 63, 65, 69, 80, 83, 84, 85], "codebas": [0, 52], "codestyl": 55, "codethink": [41, 42], "coerc": 16, "coexist": 45, "coher": 16, "col": 16, "collat": 55, "collect": [12, 13, 14, 17, 19, 20, 24, 26, 35, 40, 45, 55, 68, 73, 81], "collect_result": 24, "collid": 45, "collis": [45, 48], "colon": [0, 43, 44, 63, 85], "color": [56, 63, 84], "colour": 53, "column": [16, 62, 63], "com": [0, 1, 8, 13, 26, 32, 36, 41, 42, 45, 52, 58, 59, 62, 63, 65, 69, 73, 79, 80, 82, 83, 84, 85, 87], "combin": [0, 9, 10, 14, 41, 55, 63, 71, 85], "come": [0, 10, 32, 45, 48, 55, 56, 68, 80, 85], "comfort": 0, "comm": 58, "comma": [45, 84], "command": [0, 1, 4, 8, 9, 11, 13, 14, 15, 17, 19, 20, 21, 28, 31, 32, 33, 34, 36, 38, 39, 40, 41, 42, 43, 44, 50, 51, 55, 58, 59, 60, 61, 62, 63, 64, 65, 68, 69, 70, 71, 79, 80, 81, 86, 87, 92], "command_list": 20, "commenc": [0, 85], "comment": [0, 48, 56], "commit": [32, 43, 48, 50, 56, 57], "common": [8, 12, 13, 16, 17, 21, 25, 31, 36, 38, 39, 43, 48, 53, 55, 62, 75, 76, 77, 85], "common_config_kei": [11, 13, 21], "commonli": [69, 71, 83], "commun": [8, 30, 45, 49, 53, 84, 85], "compar": [24, 45, 48], "compat": [0, 45, 85], "compil": [5, 31, 32, 36, 45, 58, 63, 79, 80, 83, 87], "compinit": 69, "complet": [2, 7, 8, 9, 10, 14, 21, 31, 32, 36, 40, 43, 44, 45, 47, 48, 53, 56, 58, 65, 72, 73, 74, 85], "completionsdir": 69, "complex": [0, 2, 25, 44, 45, 48], "complianc": [33, 34, 35, 38, 39, 45, 85], "complic": 48, "compon": [0, 4, 24, 31, 40, 43, 45, 48, 55, 56, 65, 69, 82, 83, 85], "compos": [0, 4, 5, 14, 30, 31, 45, 47, 50, 58, 59, 60, 79, 84, 86, 87], "composit": [15, 28, 33, 43, 45, 47, 59, 60, 71, 73, 79, 80, 90], "compound": 44, "comprehens": 48, "compress": [16, 72, 84], "compris": [21, 56, 63, 83], "comput": [12, 13, 14, 17, 20, 58, 60], "compute_manifest": [11, 14], "concept": [56, 58, 61, 63, 80], "conceptu": 4, "concern": [5, 8, 9, 12, 13, 17, 20, 63, 85], "concert": 86, "concis": [50, 56], "concret": 4, "concurr": 85, "condit": [4, 15, 33, 34, 35, 38, 39, 43, 45, 48, 57, 59, 80, 85], "condition": [40, 80], "conf": [4, 9, 12, 15, 17, 21, 22, 31, 32, 36, 42, 43, 44, 45, 46, 58, 60, 62, 70, 84, 85], "config": [11, 12, 13, 14, 16, 21, 22, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 44, 45, 54, 58, 59, 60, 62, 63, 64, 65, 69, 73, 79, 80, 81, 82, 83, 84, 85, 86], "configur": [2, 3, 4, 9, 11, 13, 14, 15, 20, 21, 22, 25, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 43, 46, 48, 55, 57, 58, 59, 60, 62, 64, 65, 67, 68, 69, 71, 75, 76, 77, 79, 80, 81, 82, 83, 84], "configure_depend": [11, 12, 14, 43], "configure_sandbox": [11, 14, 20], "configurt": 68, "conflict": [15, 36, 45, 60], "conflicting_junct": [11, 15], "conform": 19, "conjunct": [5, 31], "connect": [36, 43, 45, 85], "connector": 36, "consciou": 41, "consensu": 0, "consequ": [10, 13, 31, 40, 48, 56, 84], "consid": [0, 5, 16, 17, 21, 24, 34, 44, 45, 46, 48, 55, 69, 73, 82, 85], "consider": [48, 56], "consist": [14, 16, 17, 21, 25, 26, 31, 44, 50, 55, 56, 68, 84, 85], "consol": [9, 17, 85], "const": [31, 59], "constant": [45, 48], "constantli": 63, "constitut": 48, "constraint": [16, 73], "construct": [12, 13, 17, 20, 44, 78, 79, 80], "constructor": 48, "consult": [45, 69, 73, 84], "consum": [2, 10, 21, 31, 36, 43, 55, 63, 64, 83], "contain": [0, 1, 2, 4, 8, 9, 12, 14, 15, 16, 17, 19, 24, 30, 34, 36, 43, 44, 45, 46, 51, 53, 54, 56, 58, 60, 62, 63, 68, 69, 73, 77, 79, 84, 85, 87, 89], "containeris": 0, "contamin": [4, 21], "content": [8, 9, 12, 13, 14, 16, 17, 19, 20, 21, 31, 40, 44, 55, 59, 65, 66, 69, 80, 85, 86], "contentaddressablestorag": 86, "context": [12, 13, 17, 19, 20, 21, 24, 26, 44, 45, 47, 48, 53, 58, 63, 66, 68, 85, 88], "continu": [8, 48, 55, 84, 85], "contract": [0, 48, 63], "contrast": 45, "contrib": 1, "contribut": [45, 50, 61], "contributor": [0, 50, 56], "control": [0, 9, 32, 43, 45, 48, 54, 62, 65, 69, 71, 84, 87], "convei": 45, "conveni": [4, 12, 13, 14, 16, 17, 20, 36, 40, 43, 45, 63, 81], "convert": 16, "convinc": [36, 63], "cooper": [45, 85], "copi": [9, 13, 16, 21, 26, 33, 34, 35, 38, 39, 45, 50, 54, 56, 68, 69, 84, 85], "copy2": 26, "copy_fil": [11, 26, 29, 67], "copystat": 26, "core": [0, 2, 4, 14, 17, 21, 22, 24, 25, 26, 29, 31, 32, 41, 42, 44, 45, 46, 48, 53, 58, 59, 60, 62, 63, 64, 67, 68, 69, 79, 80, 81, 82, 83, 84, 85], "core_plugin": 57, "coreutil": 40, "corewarn": [11, 25, 29, 67], "correct": [2, 9, 26, 31, 50, 54, 87], "correctli": [16, 55, 87], "correspond": [13, 24, 32, 46, 49, 54, 84, 85], "corrupt": [9, 45], "cost": [2, 14], "could": [2, 12, 13, 15, 16, 17, 20, 21, 26, 48, 55, 56, 60, 69, 81, 83], "count": [48, 80], "coupl": [45, 50, 63, 87], "cours": [0, 21, 31, 48, 50, 60], "cov": 55, "cover": [31, 32, 34, 44, 51, 58, 61, 88, 90, 91], "coverag": [54, 57], "cp": [44, 69], "cp36": 54, "cpack": 58, "cpio": 58, "cprofil": 52, "cpu": 10, "cpython": 54, "crack": 0, "creat": [0, 9, 12, 14, 16, 19, 20, 21, 24, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 44, 45, 46, 47, 48, 50, 51, 53, 55, 56, 58, 59, 60, 63, 64, 65, 71, 83, 84, 85, 89, 92], "creation": [84, 85], "credenti": [73, 84, 85], "criterion": 0, "crond": 58, "crontab": 58, "cross": [24, 26, 36, 45, 46, 56, 63, 72, 84], "crt": [45, 73, 85], "crucial": 87, "cryptpw": 58, "ctest": 58, "curl": 1, "curli": 85, "current": [2, 9, 12, 14, 16, 17, 21, 24, 25, 45, 48, 50, 52, 69, 84, 85], "custom": [12, 14, 15, 22, 28, 29, 36, 38, 43, 44, 58, 63, 65, 68, 80], "customel": 44, "customfilenam": 76, "cut": [48, 58], "cwd": [12, 14, 17, 19, 20, 39], "cycl": [0, 31, 32, 65], "cyclic": [15, 43, 48], "cython": [52, 55, 69], "d": [31, 41, 42, 59, 60, 79, 80, 83, 84], "d0b38561afb8122a3fc6bafc5a733ec502fcaed6": [36, 43], "d8": 73, "d83e1f0": 86, "daemon": 82, "dai": [48, 56], "damag": 9, "darwin": 45, "dash": 45, "dat": 58, "data": [0, 2, 3, 5, 11, 12, 14, 15, 16, 17, 20, 21, 22, 24, 26, 28, 31, 34, 35, 43, 44, 45, 48, 55, 58, 60, 65, 66, 67, 69, 73, 84, 85, 86], "datadir": [44, 45, 58, 59, 62, 63, 79], "datafil": 55, "date": 50, "db2ce036": 59, "dbus_session_bus_address": 45, "dc": 58, "dcasd": 69, "ddefault_person": 59, "deactiv": 69, "deal": [4, 60], "deallocvt": 58, "deb": 73, "debian": [55, 65, 69], "debug": [11, 12, 17, 36, 44, 45, 73, 79, 83, 84, 85], "debug_el": 45, "debugdir": [45, 79], "decid": [10, 21, 44, 45, 47, 56, 59, 63, 80, 84, 85], "decis": [0, 44, 48, 63], "declar": [4, 10, 12, 13, 14, 15, 21, 28, 30, 31, 34, 36, 40, 45, 46, 48, 54, 56, 59, 62, 63, 64, 65, 67, 68, 79, 80, 82, 83, 84, 85, 86], "decod": 48, "decor": [48, 55, 56], "deep": [0, 48], "deepli": [36, 45], "def": [13, 21, 48], "default": [2, 4, 9, 12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 26, 28, 31, 32, 33, 34, 35, 36, 38, 39, 41, 42, 43, 48, 53, 55, 56, 58, 59, 63, 69, 71, 72, 73, 74, 77, 79, 80, 83, 84, 86, 87], "default_mirror": 84, "default_person": 59, "defect": 0, "defer": 19, "defin": [2, 3, 4, 12, 14, 16, 17, 20, 21, 22, 25, 33, 34, 36, 39, 43, 44, 45, 48, 53, 54, 55, 56, 58, 59, 63, 64, 65, 68, 76, 77, 79, 80, 83, 85, 87], "definit": [2, 44, 45, 52, 65], "degre": [0, 87], "del": 16, "deleg": [9, 48], "delet": [16, 24, 26, 79, 85], "delgroup": 58, "deliber": 9, "delimit": 55, "deliv": [17, 65], "deliveri": 65, "delus": 58, "demand": [21, 36, 84, 85], "demonstr": [0, 43, 56, 59, 80], "denot": [48, 58], "dep": [40, 41, 42, 53, 69, 72, 79, 84, 85], "depend": [0, 2, 3, 4, 9, 11, 14, 15, 20, 21, 28, 30, 31, 32, 33, 34, 36, 39, 44, 45, 46, 47, 48, 49, 56, 57, 58, 59, 60, 62, 63, 64, 65, 66, 68, 72, 73, 77, 79, 80, 81, 82, 83, 84, 85, 87, 91], "dependency_path": 20, "dependencyconfigur": [11, 12, 14, 29, 67], "depends_on": 87, "depict": 24, "deploi": [5, 58, 65], "deploy": [33, 40, 44, 47, 82], "deprec": [14, 17], "depth": [10, 43], "deriv": [9, 10, 12, 13, 14, 17, 20, 21, 22, 40, 47, 48, 56, 84], "describ": [0, 3, 4, 8, 9, 12, 14, 15, 16, 21, 25, 26, 43, 44, 47, 50, 56, 58, 62, 65, 68, 80, 83, 85, 89], "descript": [0, 17, 21, 22, 24, 26, 31, 45, 48, 58, 59, 63, 79, 80, 82, 83, 84], "design": [4, 31, 48, 63, 65], "desir": [13, 21, 36, 43, 48, 60, 69, 85], "dest": [14, 24, 26], "destdir": [12, 31, 32, 58, 59, 60, 79, 80, 82, 83], "destin": [24, 26, 38, 73], "destination_dir": 24, "detail": [0, 2, 4, 9, 12, 14, 17, 19, 21, 22, 28, 36, 38, 39, 43, 45, 48, 52, 56, 61, 66, 67, 68, 73, 74, 75, 76, 77, 80, 82, 83, 84, 85, 86], "detect": [15, 25, 32, 36, 48, 60], "determin": [10, 16, 21, 34, 48, 69, 73, 85, 87], "determinist": [14, 20, 21, 25, 31, 45], "detil": 25, "detox": 55, "dev": [9, 50, 55, 69, 79], "dev0": 50, "devel": [44, 45, 55, 58, 69], "develop": [0, 5, 31, 32, 45, 48, 50, 52, 55, 56, 63, 64, 65, 69, 71, 84, 85, 87], "deviat": [4, 48, 56], "devic": [17, 26], "devlop": 32, "df": 84, "df66a655": 63, "df66a655e4245f317fd4f2784982198759448aea207966923e5d27c3d41082b0": 63, "dfuse": 69, "dgreeting_messag": 80, "diagram": 4, "dict": [13, 14, 16, 17, 19, 21, 22, 25], "dict_item": 16, "dict_valu": 16, "dictat": [0, 25, 87], "dictionari": [4, 12, 13, 14, 15, 16, 17, 19, 20, 22, 36, 43, 44, 45, 46, 56, 62, 73, 80, 85], "did": [15, 26, 31, 48, 80], "diff": 58, "differ": [0, 2, 8, 12, 14, 15, 19, 21, 26, 31, 36, 43, 44, 45, 50, 56, 58, 59, 63, 69, 80, 82, 83, 85, 87], "difficult": [48, 56], "dig": 48, "digest": [3, 24, 56], "digit": [14, 45], "digitaloceanspac": [32, 58, 59, 62, 63, 79, 80, 82, 83], "dim": 31, "dir": [20, 32, 41, 42, 77, 84, 85], "direct": [4, 12, 14, 15, 28, 45, 46, 58, 63, 71, 92], "directli": [0, 16, 17, 21, 30, 36, 45, 48, 50, 55, 62, 64, 68, 69, 73, 77, 78, 79, 81, 84], "directori": [3, 4, 9, 11, 13, 14, 15, 17, 19, 20, 21, 23, 25, 26, 28, 29, 30, 31, 32, 35, 39, 40, 43, 45, 46, 49, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 63, 64, 67, 68, 69, 71, 72, 73, 77, 79, 80, 81, 82, 83, 84], "directoryerror": [11, 23, 24, 29, 67], "directoryexistserror": [11, 26, 29, 67], "dirnam": 58, "disabl": [2, 31, 36, 44, 50, 55, 59, 84, 85], "disagre": 45, "disallow": 45, "disambigu": [48, 83], "discard": [32, 60], "discov": [17, 51, 59, 60, 69, 85], "discover": 56, "discoveri": 4, "discret": 48, "discuss": [0, 13, 21, 36, 43, 48, 56, 58, 60, 63, 69], "disk": [26, 85], "dispatch": [10, 48], "displai": [16, 17, 45, 48, 55, 72, 79, 85], "dist": 50, "distinct": [5, 48, 56, 68, 85], "distinguish": 45, "distract": 56, "distribut": [9, 31, 32, 33, 34, 35, 38, 39, 43, 45, 50, 51, 55, 56, 58, 59, 60, 62, 63, 64, 65, 73, 79, 80, 81, 82, 83, 85], "distro": 69, "distutil": [65, 73], "dnf": 55, "do": [0, 2, 4, 12, 14, 16, 17, 21, 25, 32, 36, 39, 40, 43, 44, 45, 48, 50, 52, 53, 54, 55, 56, 57, 58, 66, 69, 79, 81, 83, 84, 85], "doc": [31, 32, 45, 50, 51, 52, 55, 57, 58, 59, 60, 61, 62, 63, 64, 79, 80, 81, 82, 83], "docdir": [45, 60, 79], "docker": [27, 67, 86, 87], "docstr": [48, 50, 56], "doctor": 83, "document": [0, 1, 2, 14, 16, 17, 25, 34, 36, 38, 39, 41, 42, 43, 44, 45, 50, 52, 55, 58, 65, 67, 68, 70, 71, 72, 73, 74, 79, 81, 83, 84, 86, 89], "doe": [0, 1, 8, 10, 12, 14, 15, 17, 19, 20, 21, 25, 34, 38, 45, 48, 50, 56, 58, 60, 63, 64, 83, 84, 85], "doesn": [1, 2, 14, 16, 21, 25, 69], "doesnt": [45, 81], "domain": [10, 14, 26, 33, 34, 43, 44, 45, 47, 55, 58, 59, 60, 84], "don": [0, 2, 24, 43, 48, 50, 55, 56, 58, 68, 85], "done": [0, 4, 10, 12, 14, 16, 20, 45, 48, 50, 55, 56, 57, 63, 69, 73, 79, 83], "dont": 26, "dos2unix": 58, "doubl": [43, 48], "doument": [75, 76, 77], "down": [0, 25, 43, 83, 86, 87], "download": [1, 10, 11, 17, 21, 29, 42, 45, 50, 56, 67, 69, 72, 76, 77, 79, 83, 84, 85, 86], "downloadablefilesourc": [11, 29, 67], "downstream": [36, 45, 50, 63, 64, 86], "dozen": 55, "dpkg_build": 43, "driver": 87, "drop": [2, 9, 36, 54], "drp0a3w0": 83, "drun_bubblewrap": 69, "dtool": 69, "du": 58, "due": [0, 9, 26, 32, 36, 45, 55, 57, 72, 83, 85], "dumb": 45, "dump": 84, "dumpleas": 58, "dunder": 48, "duplic": [50, 68], "durat": [17, 55], "dure": [4, 9, 15, 21, 26, 52, 68], "dwp": 58, "dymam": 31, "dynam": [14, 43], "e": [0, 2, 3, 15, 17, 21, 31, 32, 36, 41, 42, 43, 45, 46, 47, 48, 54, 55, 56, 59, 65, 69, 82, 84, 85], "e2": 73, "each": [0, 2, 4, 9, 10, 20, 33, 34, 43, 44, 46, 47, 52, 54, 55, 56, 65, 68, 80, 84, 85], "eachoth": [25, 60, 62, 63, 91], "earli": [2, 17], "earlier": [45, 63], "earn": 0, "earth": [45, 85], "easi": [50, 53, 56], "easier": [48, 55], "easiest": [50, 60], "easili": [0, 36, 40, 50, 56, 59, 64], "easy_instal": 58, "eb9542a1a6a86e71e839554cf67aca2e2fa52c29eefbb8707889bae8ebd9d429": 60, "ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5": 31, "echo": [62, 79], "ed9e849a1386297f854f9fa0200f3fa108498c0fdb5c86468c1601c7e571": 73, "edg": [57, 59], "edit": [1, 31, 32, 45, 56, 69], "eexist": 26, "effect": [0, 4, 14, 16, 21, 25, 31, 43, 48, 50, 59, 63, 79, 85], "effort": [48, 85], "either": [2, 4, 12, 13, 14, 15, 16, 20, 21, 24, 26, 33, 34, 35, 38, 39, 45, 48, 59, 63, 84, 85], "eject": 58, "ejrdsw3w": 80, "elabor": [55, 67], "elaps": 85, "elect": 46, "eleg": 56, "element": [0, 1, 2, 3, 5, 10, 11, 13, 15, 16, 17, 19, 21, 24, 25, 28, 29, 32, 35, 39, 41, 42, 44, 46, 47, 48, 52, 53, 56, 57, 60, 64, 65, 66, 67, 68, 69, 71, 72, 78, 84, 85, 86, 87, 91, 92], "element1": [43, 44], "element2": [43, 44], "element_path": 84, "elementerror": [11, 12, 13, 14, 17, 20, 24, 26, 29, 48, 67], "elementjob": 10, "elfedit": 58, "elif": [17, 62, 63, 79], "elimin": 85, "els": [14, 16, 17, 41, 59, 62, 63, 79], "elsewher": [48, 63], "email": 50, "emb": 2, "embed": 2, "emit": [17, 46], "emphas": 63, "emploi": 5, "empti": [0, 12, 14, 24, 25, 26, 33, 34, 35, 38, 45, 56, 77, 84, 85], "en": 55, "enabl": [2, 9, 31, 32, 36, 41, 42, 43, 44, 45, 54, 55, 56, 58, 59, 60, 62, 63, 64, 65, 79, 80, 81, 82, 83, 84, 85, 86], "encapsul": [16, 48, 63], "enclos": 85, "encod": [15, 24, 26, 48], "encount": [15, 21, 43, 45, 48, 84], "encourag": [45, 76, 77, 83], "encrypt": 85, "end": [12, 14, 15, 19, 26, 33, 48, 55, 56, 64, 69, 80, 84, 85], "endif": 59, "endpoint": 85, "enforc": [48, 63], "enotempti": 26, "enough": [48, 54, 56, 58], "ensu": 0, "ensur": [4, 9, 12, 13, 16, 17, 19, 21, 26, 31, 32, 36, 45, 48, 50, 52, 54, 56, 57, 59, 73, 82, 83, 85], "ensure_par": 26, "entir": [0, 9, 31, 45, 48, 59, 63, 85, 87], "entiti": 85, "entitl": 8, "entri": [4, 13, 16, 17, 24, 25, 50, 53, 56, 57, 85], "entry_point": 17, "enum": [15, 16, 25, 45, 63, 80], "enumer": 14, "env": [17, 19, 56, 58, 84, 85], "environ": [1, 2, 4, 5, 8, 9, 12, 13, 14, 17, 19, 20, 30, 41, 44, 47, 50, 54, 55, 56, 65, 72, 83, 84, 85, 87], "envlist": 54, "equal": [12, 13, 17, 20, 24, 34, 40, 45, 85], "equip": 79, "equival": [16, 24, 43], "error": [0, 8, 11, 12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 25, 26, 29, 33, 36, 44, 45, 55, 60, 67, 84, 85], "error_lin": 84, "errordomain": [11, 15, 29, 67], "escap": [24, 26], "especi": [0, 26, 45, 48, 82, 85], "essenti": [28, 43, 50, 59, 63, 85], "etc": [0, 4, 9, 15, 43, 45, 47, 48, 62, 63, 68, 79, 85, 87], "ether": 58, "evalu": [24, 44, 45, 65], "even": [5, 17, 19, 31, 36, 40, 43, 45, 48, 84, 85], "event": 10, "event_engine_dn": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "event_engine_listen": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "ever": [43, 48], "everi": [0, 10, 14, 16, 21, 24, 26, 48, 54, 55, 56, 57, 82, 85], "everyon": 0, "everyth": [4, 16, 25, 35, 48, 50, 51, 61, 83, 85], "evolv": 48, "exact": [9, 10, 12, 13, 17, 20, 21, 25, 45, 55], "exactli": [4, 12, 13, 16, 17, 20, 22, 34, 45, 48, 83], "examin": [55, 82], "exampl": [0, 1, 2, 5, 8, 10, 12, 13, 14, 16, 17, 21, 22, 26, 31, 32, 34, 36, 39, 40, 41, 42, 43, 44, 45, 46, 48, 50, 52, 54, 55, 59, 60, 63, 64, 71, 79, 80, 81, 82, 83, 84, 85], "exceed": 85, "excel": 55, "except": [4, 9, 11, 14, 16, 17, 19, 20, 21, 22, 24, 26, 29, 31, 33, 34, 35, 38, 39, 43, 44, 45, 59, 62, 63, 67, 72, 82, 83, 84, 85], "except_": 84, "excess": 55, "exchang": [63, 86], "excit": 80, "exclam": 50, "exclud": [5, 14, 22, 33, 34, 58, 59, 60, 84], "exclus": [2, 10, 41, 42], "exec": [62, 63, 79], "exec_prefix": [45, 79], "execut": [1, 3, 9, 11, 12, 14, 17, 19, 20, 23, 24, 25, 45, 52, 55, 66, 71, 76, 83, 84, 86], "exercis": [55, 56, 79, 82], "exist": [0, 9, 10, 11, 12, 14, 16, 17, 19, 21, 23, 24, 26, 28, 32, 33, 34, 36, 41, 44, 45, 48, 50, 54, 55, 56, 57, 58, 60, 65, 69, 70, 72, 73, 84, 85], "exit": [12, 14, 17, 19, 85], "exit_cod": 17, "exlus": 34, "expand": [12, 58, 59, 85], "expans": [45, 58, 84], "expect": [0, 5, 9, 10, 12, 13, 14, 15, 16, 17, 20, 21, 25, 31, 36, 43, 44, 45, 47, 48, 57, 59, 63, 73, 81, 82, 85, 89], "expens": [9, 85], "experi": [31, 32, 55, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "experienc": 0, "experiment": [30, 73], "expiri": [0, 68], "explain": [0, 14, 31, 43, 44, 45, 48, 56, 63, 69, 84], "explan": [43, 48, 56, 73], "explicit": [56, 73], "explicitli": [12, 14, 21, 22, 31, 36, 45, 48, 54, 56, 58, 59, 60, 73, 85], "explor": [31, 58, 59, 63, 65, 80, 82, 83], "export": [12, 14, 24, 45, 56, 62, 63, 69, 79], "export_to_tar": [11, 23, 24], "expos": [4, 9, 20, 21, 22, 47, 48, 52, 62, 63, 83, 85], "expr": 58, "express": [4, 15, 17, 22, 33, 34, 35, 36, 38, 39, 44, 45, 48, 62, 73, 84, 85], "expression_fail": [11, 15], "extend": [9, 17, 58, 60, 65], "extens": [12, 13, 14, 17, 20, 43, 55, 65, 66, 73, 82, 84], "extensibli": 65, "extern": [4, 15, 17, 21, 24, 45, 48, 55, 62, 67, 68, 69, 79, 85], "external_directori": 24, "external_path": 24, "external_pathspec": 24, "extra": [9, 11, 16, 22, 44, 45, 69, 84, 85], "extra_data": [13, 21, 22], "extra_flag": 59, "extract": [2, 16, 30, 48, 50, 56, 69, 77], "extracted_artifact": 48, "f": [26, 31, 41, 42, 59, 60, 84, 86, 87], "f15gxvq4": 63, "face": [24, 45, 48, 50, 53, 71], "facet": 31, "facil": 48, "factor": [9, 58], "factori": 4, "fail": [9, 12, 14, 15, 17, 19, 24, 26, 31, 32, 34, 43, 45, 48, 55, 58, 59, 60, 62, 63, 72, 79, 80, 81, 82, 83, 84, 85], "fail_temporarili": 17, "failed_attribut": [11, 26], "failur": [15, 17, 19, 26, 48, 55, 57, 85], "fairli": 63, "fake": 56, "fall": [21, 48, 85, 86], "fallback": [84, 85], "falloc": 58, "fals": [14, 16, 17, 19, 21, 22, 24, 26, 34, 39, 43, 44, 45, 48, 58, 59, 63, 79, 85], "falsi": 24, "familiar": [58, 92], "familliar": 80, "fanci": [43, 47], "faq": 69, "far": [10, 25, 81], "fashion": 55, "fast": [10, 65], "fastenum": [11, 16, 24, 25, 29, 67], "faster": [8, 48, 65], "fatal": [17, 25, 45, 60], "favor": [72, 73, 74], "favourit": 65, "fbset": 58, "fdformat": 58, "featur": [9, 12, 17, 32, 36, 43, 45, 47, 48, 50, 56, 57, 59, 63, 65, 73, 84, 85, 88, 91], "fedora": [55, 69], "feel": [0, 56, 81], "fell": 0, "fetch": [3, 5, 10, 11, 13, 14, 17, 19, 25, 29, 31, 32, 36, 45, 56, 58, 60, 62, 63, 64, 65, 67, 69, 71, 79, 80, 81, 82, 83, 86], "fetcher": [21, 84, 85], "few": [0, 9, 10, 43, 44, 48, 55, 56, 58, 80], "ffeab612": [62, 64, 79], "ffeab61203b5940714c33b1cc29822b9391b9fea5876faef23c024bf6acd99cf": [62, 79], "fi": [59, 62, 63, 79], "field": [21, 34, 43, 45, 85], "fifo": 17, "file": [3, 4, 8, 9, 11, 12, 13, 14, 15, 16, 17, 21, 22, 23, 25, 26, 28, 29, 30, 32, 33, 35, 38, 39, 40, 42, 43, 44, 47, 49, 50, 52, 54, 55, 56, 57, 58, 62, 63, 64, 65, 67, 68, 69, 71, 73, 74, 79, 81, 82, 84], "file_digest": [11, 23, 24], "file_typ": [11, 23, 24], "filelistresult": [11, 14, 24, 26, 29, 67], "filenam": [15, 17, 24, 26, 31, 39, 43, 44, 45, 48, 62, 63, 76], "filenotfounderror": 48, "files_written": [11, 26], "filestat": [11, 23, 24, 29, 67], "filesystem": [4, 8, 14, 20, 24, 26, 58, 65, 85], "filetyp": [11, 23, 24, 29, 67], "filt": 58, "filter": [5, 24, 26, 30, 60, 71, 90], "filter_callback": [24, 26], "final": [4, 31, 43, 44, 50, 58, 69, 79, 80, 83], "find": [45, 48, 50, 55, 58, 63, 73, 79, 82, 85, 86], "find_packag": 17, "fine": [43, 44], "finer": 90, "finish": [8, 81], "first": [0, 4, 10, 14, 21, 31, 32, 43, 44, 45, 47, 48, 50, 55, 56, 58, 59, 63, 65, 69, 71, 77, 80, 82, 83, 84, 85, 92], "five": 80, "fix": [31, 50, 55], "fixup": 0, "flag": [19, 53, 63, 69, 84, 85], "flalback": 85, "flat": 3, "flatpak": 65, "flavor": 80, "flexibl": [22, 65, 73, 80], "float": [24, 44], "flock": 58, "flow": 66, "fly": 13, "fmt": 17, "focal": 4, "focu": [31, 56, 80], "focus": [56, 70], "fold": 58, "folder": [38, 63, 79], "folk": 0, "follow": [0, 5, 9, 10, 12, 13, 14, 17, 21, 22, 24, 26, 31, 32, 34, 39, 41, 42, 43, 45, 48, 50, 51, 53, 54, 55, 56, 60, 65, 69, 73, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87], "follow_symlink": 24, "font": 82, "fontcach": 47, "foo": [0, 17, 22, 26, 34, 39, 43, 45, 48, 56, 58, 73, 76, 77, 85], "fooerror": 48, "forbidden": 14, "forc": [32, 55, 56, 84], "forcefulli": [5, 10, 32], "fork": [36, 63], "form": [0, 4, 8, 16, 17, 26, 44, 45, 50, 56, 63, 68, 82, 83, 84], "formal": 0, "format": [0, 12, 14, 16, 17, 39, 44, 45, 47, 48, 50, 52, 53, 65, 67, 68, 70, 72, 79, 80, 84, 85], "former": 48, "formerli": 30, "forward": [0, 34, 43, 48, 59, 84], "found": [0, 12, 14, 15, 16, 17, 19, 21, 24, 25, 26, 35, 36, 45, 46, 48, 50, 55, 72, 73, 84, 85], "foundat": [11, 14, 29, 67], "four": [83, 85], "fpath": 69, "fragil": 55, "fragment": [44, 45, 63], "framework": [4, 65, 69], "free": [0, 4, 44, 56, 58, 82, 85], "freebsd": 45, "freedesktop": [12, 62, 65], "freedom": 48, "frequent": 31, "fresher": 45, "friction": 63, "frob": [0, 43, 48], "frobbish": 48, "frobdir": 43, "froberror": 48, "frobilici": 48, "frobnic": [0, 43, 45, 48], "from": [0, 1, 2, 4, 8, 9, 10, 11, 12, 14, 15, 16, 17, 20, 21, 22, 24, 25, 26, 29, 30, 31, 32, 33, 35, 36, 39, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 56, 58, 59, 60, 61, 62, 64, 65, 67, 68, 72, 73, 79, 80, 81, 82, 83, 84, 85, 86, 87], "from_dict": [11, 16], "frontend": [48, 53, 55, 56, 69], "frozen": 50, "frustrat": [0, 63], "ftp": 42, "ftp_gnu_org": 42, "ftpadmin": 50, "ftpmirror": [62, 63, 79], "full": [0, 8, 9, 14, 17, 26, 45, 52, 57, 59, 65, 69, 84, 85], "fulli": [2, 9, 12, 13, 17, 20, 21, 43, 45, 56], "function": [2, 5, 8, 10, 11, 13, 14, 17, 19, 20, 24, 25, 26, 30, 32, 38, 43, 47, 48, 52, 55, 72, 75, 76, 77, 86], "fundament": [83, 92], "further": [5, 12, 14, 40, 43, 48, 56], "furthermor": 63, "fuse": [9, 69], "fuse3": 69, "fuser": 58, "fusermount3": 69, "futur": [0, 9, 12, 13, 17, 20, 36, 48], "g": [0, 2, 3, 15, 21, 36, 41, 42, 43, 45, 48, 55, 56, 65, 69, 84, 85], "g65e2074ad": [31, 32, 58, 60, 62, 63, 79, 80, 81, 82, 83], "gain": 0, "gcc": [36, 55, 69], "gdbm_dump": 58, "gdbm_load": 58, "gdbmtool": 58, "gener": [0, 2, 6, 7, 11, 12, 14, 17, 19, 24, 26, 27, 43, 45, 48, 50, 53, 54, 55, 56, 57, 68, 69, 73, 81, 83, 85], "generate_script": [11, 12, 14], "get": [0, 1, 2, 10, 13, 14, 16, 17, 21, 24, 25, 26, 32, 45, 48, 50, 52, 55, 56, 71, 80, 81, 85], "get_": 48, "get_artifact_nam": [11, 14], "get_bool": [11, 16], "get_bst_vers": [11, 26, 29, 67], "get_count": 48, "get_enum": [11, 16], "get_environ": [11, 14], "get_foo": 48, "get_host_tool": [11, 12, 13, 17, 20, 26, 29, 67], "get_int": [11, 16], "get_kind": [11, 17], "get_map": [11, 16], "get_mirror_directori": [11, 21], "get_nod": [11, 16], "get_project_directori": [11, 21], "get_proven": [11, 16], "get_public_data": [11, 14], "get_ref": [11, 13, 21], "get_scalar": [11, 14, 16], "get_sequ": [11, 16], "get_source_fetch": [11, 21], "get_str": [11, 16], "get_str_list": [11, 16], "get_umask": [11, 26, 29, 67], "get_unique_kei": [2, 3, 11, 12, 13, 17, 20, 21, 57], "get_vari": [11, 12, 14], "get_virtual_directori": [11, 18, 19, 24], "get_workspac": 48, "getconf": 58, "getent": 58, "gettext": 42, "gid": [9, 43, 45], "git": [0, 17, 21, 36, 42, 43, 45, 48, 49, 50, 62, 65, 71, 73, 89], "git_gnome_org": 41, "git_http_export_al": 41, "git_project_root": 41, "github": [0, 8, 50, 65, 69], "githubusercont": 1, "gitlab": [0, 50, 52, 65, 69, 87], "give": [0, 9, 12, 14, 32, 43, 44, 48, 52, 55, 56, 69, 81], "given": [2, 4, 5, 10, 12, 13, 14, 16, 17, 19, 21, 22, 24, 30, 33, 35, 40, 43, 44, 45, 46, 47, 48, 50, 56, 58, 59, 60, 63, 72, 73, 79, 84, 85, 87], "glibc": [36, 54], "glob": [11, 26, 29, 47, 54, 58, 60, 67, 77], "global": [17, 72, 79, 83], "glossari": 61, "gnome": [41, 42, 50, 62, 65], "gnome7": [41, 42], "gnu": [42, 54, 62, 63, 79], "go": [12, 13, 17, 20, 31, 43, 44, 45, 48, 50, 53, 56, 60, 63, 79, 80, 81, 83, 84], "goal": 63, "gone": 63, "good": [0, 12, 13, 17, 20, 31, 48, 50, 79], "googl": [3, 85, 87], "gotten": 58, "govern": [0, 33, 34, 35, 38, 39, 45, 85], "gprof": 58, "gracefulli": 48, "grain": 90, "granular": 5, "graph": [5, 14, 36, 40, 65, 85], "great": 4, "greater": 43, "green": 63, "greet": 31, "greeting_messag": 80, "grep": 79, "grid": 87, "group": [9, 19, 20, 30, 43, 45, 53, 55, 58, 83], "group_nam": 20, "grpc": [31, 32, 49, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 85], "grpcio": 49, "guarante": [2, 9, 12, 13, 14, 17, 20, 21, 36, 44, 45, 48, 55, 87], "guard": 63, "guess": [44, 50], "guid": [61, 65, 68, 71], "guidelin": [0, 21], "gz": [50, 62, 63, 72, 73, 79, 84], "gzip": 16, "h": [31, 32, 58, 59, 60, 80, 83, 84], "ha": [0, 2, 3, 4, 10, 12, 13, 14, 15, 16, 17, 20, 21, 31, 32, 41, 43, 44, 45, 46, 47, 48, 50, 52, 53, 54, 56, 58, 59, 60, 63, 64, 68, 72, 73, 74, 79, 80, 81, 84, 85, 87], "hack": [0, 48], "had": [0, 8], "hand": [48, 56, 63], "handl": [10, 11, 13, 16, 24, 25, 26, 29, 53, 60, 65, 67, 71, 82], "hang": 48, "happen": [4, 14, 15, 25, 26, 43, 48, 55, 60, 63, 79, 85], "hard": [32, 73, 85], "harder": 52, "hardlink": [9, 26, 84], "hardwar": 9, "harm": 0, "hash": [2, 3], "hassl": 63, "have": [0, 1, 2, 3, 4, 5, 8, 9, 10, 12, 13, 14, 16, 17, 20, 21, 22, 24, 26, 31, 32, 34, 36, 40, 41, 42, 43, 44, 45, 48, 50, 55, 56, 57, 58, 59, 60, 62, 63, 65, 68, 69, 72, 73, 74, 79, 80, 81, 82, 83, 84, 85, 87], "haven": [45, 79, 85], "hd": 58, "he": 84, "head": [56, 58], "header": [13, 32, 45, 58, 59, 69], "hei": 80, "held": 16, "hello": [1, 32, 36, 43, 56, 58, 62, 64, 84], "help": [10, 31, 36, 45, 48, 50, 52, 53, 56, 63, 65, 69, 83, 84, 85, 92], "helper": [16, 48, 55], "henc": [43, 68], "here": [0, 1, 4, 7, 9, 12, 13, 16, 17, 20, 21, 26, 30, 31, 35, 36, 42, 43, 44, 45, 46, 47, 48, 50, 52, 54, 55, 56, 58, 59, 60, 63, 69, 72, 73, 80, 81, 82, 83, 84, 85, 86], "hexdump": 58, "hh": 85, "hi": 50, "hidden": 48, "hierarchi": 24, "high": [55, 65, 85], "higher": [36, 43, 44, 48, 83, 85], "highlevel": 7, "highlight": 50, "hippocrat": 0, "histori": [0, 48, 69], "hold": [10, 41, 42, 45, 46], "home": [31, 32, 45, 50, 58, 60, 62, 63, 64, 69, 79, 80, 81, 82, 83, 85], "homedir": 69, "hope": 9, "host": [4, 9, 12, 13, 17, 19, 20, 21, 26, 31, 32, 43, 55, 58, 59, 60, 62, 63, 64, 69, 74, 77, 79, 80, 81, 82, 83, 85], "host_path": 45, "hostid": 58, "hostpath": 84, "hour": 56, "how": [1, 2, 7, 8, 9, 10, 12, 14, 17, 20, 21, 22, 25, 28, 31, 34, 36, 41, 42, 43, 44, 45, 47, 48, 50, 52, 55, 56, 58, 59, 60, 61, 62, 63, 64, 68, 69, 73, 74, 79, 80, 82, 83, 85], "howdi": 80, "howev": [0, 2, 9, 10, 13, 16, 21, 32, 36, 40, 43, 45, 48, 54, 58, 60, 63, 73, 74, 80, 83, 85], "html": [50, 52, 55, 57], "http": [0, 1, 8, 11, 26, 32, 33, 34, 35, 36, 38, 39, 42, 45, 50, 52, 55, 58, 59, 62, 63, 65, 69, 73, 79, 80, 82, 83, 84, 85, 86, 87], "http2_stats_fix": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "hub": 87, "huge": 10, "human": [48, 85], "hundr": [58, 65], "hxcfj_9g": 81, "hypothet": 73, "i": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 30, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 63, 64, 68, 69, 72, 73, 74, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 92], "i0000": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "i18n": 45, "i386": 45, "i486": 45, "iconv": 58, "id": [9, 43, 45, 55, 58], "idea": [56, 79], "ideal": [48, 85], "ident": [2, 36, 85], "identif": 45, "identifi": [0, 12, 13, 14, 17, 20, 21, 40, 43, 48, 84, 86], "ifdef": 59, "ifnam": 58, "ignor": [11, 25, 26, 31, 32, 36, 45, 46, 60, 69, 84, 85], "ignore_miss": 26, "illeg": [24, 36, 43, 48], "illegal_composit": [11, 15], "illustr": [4, 31, 48], "imag": [27, 32, 54, 58, 59, 62, 63, 65, 79, 80, 82, 83, 86, 87], "immedi": [19, 85], "immut": 48, "impact": [0, 45, 55], "imper": 10, "impl": [11, 15], "implement": [4, 8, 9, 10, 11, 13, 14, 17, 19, 20, 21, 22, 25, 29, 40, 43, 44, 45, 46, 48, 52, 53, 55, 57, 63, 68, 69, 71, 73, 83, 84, 85], "implementor": [12, 13, 17, 20, 21, 48], "impli": [33, 34, 35, 38, 39, 43, 45, 69, 85], "implicit": 83, "implicitli": [36, 43, 83], "import": [0, 1, 4, 5, 10, 12, 14, 16, 17, 21, 24, 30, 31, 32, 34, 36, 44, 50, 54, 55, 56, 57, 58, 60, 62, 63, 73, 79, 80, 81, 82, 83], "import_fil": [11, 23, 24], "import_single_fil": [11, 23, 24], "imposs": [44, 48, 85], "impract": 55, "improv": [0, 31, 32, 80], "inadvert": 48, "incas": 26, "includ": [0, 2, 3, 4, 5, 9, 14, 15, 17, 20, 21, 26, 31, 32, 33, 34, 36, 45, 47, 48, 49, 50, 51, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 65, 71, 72, 73, 79, 83, 84, 85, 87, 91], "include_package_data": 17, "includedir": [45, 62, 63, 79], "inclus": [0, 43], "incompat": [15, 45, 57, 85], "incomplet": 26, "inconsist": [17, 84], "inconveni": [31, 45, 60], "increment": 12, "inde": 16, "indent": [56, 85], "independ": [9, 21, 45], "index": [3, 16, 17, 69, 84, 85, 86], "indexerror": 16, "indic": [12, 13, 14, 17, 20, 21, 33, 34, 40, 43, 44, 45, 47, 56, 72, 73, 77, 83, 85], "indirectli": 48, "individu": [0, 11, 12, 29, 30, 44, 45, 48, 67, 84, 85], "indulg": 83, "inevit": 17, "infer": 84, "infin": 85, "influenc": [2, 63, 82], "info": [11, 17, 31, 32, 45, 58, 60, 64, 79, 85], "infodir": [45, 79], "inform": [3, 9, 12, 13, 16, 17, 20, 30, 34, 43, 44, 45, 48, 50, 56, 60, 61, 65, 69, 71, 80, 82, 84, 85], "infrastructur": [0, 43], "inher": 44, "inherit": [45, 48, 63, 73], "ini": 55, "init": 81, "init_workspac": [11, 21], "init_workspace_directori": [11, 21], "init_workspace_workspac": 21, "initi": [8, 21, 31, 32, 43, 48, 52, 53, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 84], "initialis": 3, "initializelog": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "inlin": [45, 46, 48], "input": [2, 4, 8, 9, 10, 12, 13, 14, 17, 20, 31, 35, 39, 43, 45, 56, 58, 68, 81, 83, 85], "input_root_digest": 87, "insid": [9, 10, 12, 14, 19, 20, 21, 24, 27, 32, 45, 48, 55, 73, 83], "insight": [7, 56], "inspect": [17, 40], "instal": [1, 9, 11, 14, 19, 20, 30, 31, 32, 38, 41, 42, 43, 44, 45, 49, 50, 52, 56, 58, 59, 60, 61, 62, 63, 65, 73, 79, 80, 82, 83, 85], "install_root": 20, "instanc": [16, 17, 24, 32, 43, 45, 84, 85], "instanti": 4, "instead": [0, 2, 5, 9, 14, 16, 17, 21, 24, 26, 31, 32, 40, 44, 45, 46, 48, 55, 56, 60, 62, 63, 72, 73, 79, 80, 84, 85, 87], "instruct": [1, 12, 43, 45, 50, 55, 56, 57, 65, 69, 79, 83, 85], "insuffici": [84, 85], "int": [13, 16, 17, 19, 21, 24, 25, 26, 32, 48, 59, 80, 83], "integ": 16, "integr": [1, 8, 9, 11, 12, 14, 20, 28, 30, 31, 32, 33, 34, 40, 43, 45, 50, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 65, 69, 71, 79, 80, 81, 83, 84, 88, 92], "intend": [5, 9, 21, 45, 50, 53, 63, 82, 85], "intent": [44, 45], "intention": [15, 44], "interact": [0, 8, 9, 10, 14, 21, 22, 52, 53, 62, 63, 73, 84, 85], "interchang": 84, "interdepend": 63, "interest": [26, 29, 31, 43, 46, 50, 55, 56, 73], "interfac": [9, 11, 14, 19, 21, 22, 23, 29, 48, 55, 56, 61, 67, 72], "intern": [6, 8, 11, 13, 16, 26, 43, 48, 55, 61, 66, 67], "internet": 69, "interoper": [62, 82], "interpret": 84, "interv": 85, "intim": 45, "introduc": [14, 21, 36, 44, 48, 56, 59, 83], "introduct": [28, 61, 66, 67], "invalid": [0, 15, 16, 25, 41, 42, 44, 46], "invalid_data": [11, 15], "invalid_dependency_config": [11, 15], "invalid_junct": [11, 15], "invalid_symbol_nam": [11, 15], "invalid_yaml": [11, 15], "invari": 48, "invit": 50, "invoc": [14, 25, 48, 52, 85], "invok": [9, 41, 42, 45, 48, 62, 80, 82, 83, 84, 85], "involv": [21, 55, 83, 85], "io": [17, 24, 26], "ipc": 58, "ipcrm": 58, "irc": [50, 65], "irrelev": 82, "is_cach": [11, 13, 21], "is_non": [11, 16], "is_resolv": [11, 12, 13, 17, 20, 21], "isa": [43, 45, 65, 87], "isdir": [11, 23, 24], "isfil": [11, 23, 24], "islink": [11, 23, 24], "isn": [16, 41, 42], "isol": [9, 55, 69, 83, 84, 87], "issu": [9, 14, 16, 25, 26, 45, 50, 69, 85], "item": [11, 16, 45, 47], "iter": [10, 14, 16, 21, 24, 26, 85], "its": [0, 2, 3, 4, 5, 10, 12, 13, 14, 15, 16, 17, 20, 21, 31, 32, 33, 34, 35, 36, 40, 41, 42, 43, 44, 48, 56, 59, 62, 63, 65, 68, 69, 79, 81, 82, 84, 85], "itself": [2, 3, 5, 10, 29, 43, 44, 45, 46, 48, 56, 58, 63, 73, 79, 81, 83, 84, 85], "ivf": [62, 63, 79], "j": [45, 84], "j1": [31, 32, 58, 59, 60, 79, 80, 82, 83], "java": 65, "job": [2, 8, 9, 16, 43, 45, 48, 54, 66, 79, 84, 85], "join": [48, 84], "json": [2, 50], "jsonnet": 86, "judgement": 0, "jump": 52, "junction": [14, 15, 28, 30, 37, 44, 46, 68, 71, 72, 84, 85, 91], "just": [0, 3, 4, 16, 20, 26, 31, 45, 48, 50, 55, 56, 58, 59, 60, 62, 63, 79, 80, 81, 83, 84], "k": [20, 85], "keep": [21, 48, 50, 55, 60, 74, 85], "keepal": 85, "kei": [3, 4, 10, 11, 12, 13, 14, 16, 17, 19, 20, 25, 31, 36, 40, 43, 44, 45, 46, 48, 54, 57, 65, 66, 80, 84, 85], "kernel": 9, "keyword": [13, 21], "killal": 58, "killall5": 58, "kilobyt": 85, "kind": [2, 4, 13, 15, 17, 21, 31, 33, 34, 35, 36, 37, 38, 39, 40, 44, 45, 56, 58, 59, 62, 63, 68, 73, 75, 76, 77, 78, 79, 80, 81, 82, 83, 85], "know": [0, 12, 17, 21, 40, 45, 48, 50, 79, 82, 85], "knowledg": [0, 45, 48, 83], "known": [10, 12, 21, 30, 43, 55, 68, 71, 87], "kwarg": 21, "l": [81, 84], "la": [45, 79], "label": [19, 86], "labesl": 3, "lai": 48, "land": [0, 48, 50], "languag": [33, 34, 35, 38, 39, 45, 48, 56, 65, 85], "larg": [31, 41, 55, 85], "last": [44, 50, 55, 79, 83, 85], "latenc": 85, "later": [0, 8, 10, 13, 19, 21, 32, 33, 36, 40, 44, 45, 47, 48, 58, 60, 61, 82, 83, 85], "latest": [2, 44, 50, 54, 55, 69], "latter": 48, "launch": [5, 45, 48, 52, 82], "law": [33, 34, 35, 38, 39, 45, 85], "layer": 48, "layout": [20, 73], "layout_add": [11, 20], "lc_all": 45, "ld": [14, 58], "ld_library_path": 43, "ldconfig": [31, 47, 58, 59, 60, 82], "ldd": 58, "le": 45, "lead": [15, 17, 24, 43, 45, 48, 56, 84, 85], "leak": 9, "learn": [56, 58, 68], "least": [0, 12, 13, 17, 21, 45, 50, 58, 63], "leav": [48, 82], "left": [21, 36, 45, 55], "legibl": 48, "len": 16, "length": [16, 85], "less": [4, 8, 43, 48, 52, 58], "let": [17, 31, 32, 43, 44, 48, 58, 59, 63, 79, 80, 81, 83], "letter": [0, 14, 43], "level": [4, 5, 9, 19, 26, 31, 36, 40, 44, 45, 48, 55, 71, 72, 73, 83, 85], "leverag": [48, 68], "lhello": [31, 59, 60], "lib": [31, 41, 45, 47, 58, 59, 60, 62, 63, 79, 82, 87], "libarch": 58, "libbfd": 58, "libbz2": 58, "libc": [40, 58], "libcrypto": 58, "libcurl": 58, "libdir": [44, 45, 47, 58, 62, 63, 79, 82], "libexec": [45, 62, 63, 79], "libexecdir": [45, 47, 62, 63, 79], "libexpat": 58, "libffi": 58, "libformw": 58, "libgcc_": 58, "libgdbm": 58, "libgdbm_compat": 58, "libhello": 58, "liblz4": 58, "liblzma": 58, "libmenuw": 58, "libncursesw": 58, "libopcod": 58, "libpanelw": 58, "libpython3": 58, "librari": [31, 36, 45, 48, 52, 55, 58, 59, 60, 79, 82, 83], "libreadlin": 58, "librhash": 58, "libsqlite3": 58, "libssh2": 58, "libssl": 58, "libstdc": 58, "libtool": [45, 79], "libuv": 58, "licens": [0, 33, 34, 35, 38, 39, 45, 85], "life": 48, "lifetim": [9, 48, 50], "light": 0, "like": [0, 1, 9, 16, 17, 26, 31, 32, 34, 36, 43, 44, 45, 48, 50, 55, 56, 58, 59, 62, 65, 79, 81, 83, 84, 85], "limit": [4, 9, 33, 34, 35, 36, 38, 39, 45, 48, 63, 68, 85], "line": [0, 4, 15, 16, 20, 41, 42, 43, 45, 48, 50, 56, 61, 62, 63, 69, 70, 79, 80, 84], "liner": 55, "link": [0, 15, 17, 24, 26, 30, 31, 32, 43, 50, 56, 57, 63, 69, 73, 84], "link_fil": [11, 26, 29, 67], "link_forbidden_depend": [11, 15], "linker": [58, 82], "lint": 55, "linux": [8, 12, 43, 45, 54, 58, 69, 82, 83, 85, 87], "linux32": 43, "list": [0, 3, 4, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 30, 32, 33, 34, 39, 40, 41, 42, 43, 45, 46, 47, 48, 50, 52, 54, 55, 56, 59, 68, 69, 73, 83], "list_relative_path": [11, 23, 24, 26, 29, 67], "liter": [43, 44, 83], "littl": [7, 48, 81], "live": [49, 63, 79], "ll": [10, 20, 43, 58, 60, 79, 80, 81, 82, 83], "ln": 69, "load": [4, 10, 11, 12, 13, 14, 15, 16, 17, 22, 28, 30, 31, 32, 36, 44, 48, 52, 55, 58, 59, 60, 62, 63, 64, 65, 74, 79, 80, 81, 82, 83, 85], "load_el": [12, 20], "load_ref": [11, 13, 21], "loadel": [12, 20], "loaderror": [14, 15, 16, 17], "loaderrorreason": [11, 15, 29, 67], "loadfont": 58, "loading_directori": [11, 15], "local": [0, 1, 2, 3, 4, 8, 9, 10, 13, 21, 30, 31, 32, 34, 36, 41, 42, 43, 44, 46, 48, 56, 58, 59, 60, 62, 63, 64, 65, 68, 69, 79, 80, 81, 82, 83, 84, 85, 86], "localhost": [86, 87], "localstatedir": [45, 62, 63, 79], "locat": [3, 4, 11, 13, 14, 16, 20, 21, 39, 43, 45, 56, 58, 69, 72, 73, 79, 84, 85], "lock": 54, "log": [3, 4, 8, 11, 14, 17, 19, 31, 32, 40, 44, 45, 48, 53, 58, 59, 60, 62, 63, 64, 68, 71, 73, 79, 80, 81, 82, 83], "log_fil": 84, "logdir": 85, "logger": 58, "logic": [40, 44, 58, 83], "login": 50, "loglevel": 45, "loglin": 53, "logmask": 45, "lognam": 45, "long": [2, 10, 43, 48, 58, 84], "longer": [36, 48, 60, 72, 73, 85], "look": [10, 12, 22, 31, 36, 44, 45, 52, 55, 59, 62, 63, 80, 81, 83, 84, 85], "loop": 10, "loos": 55, "lose": 48, "lost": [8, 55], "lot": [0, 9, 31, 55], "low": [4, 31, 48, 55, 85], "lower": [31, 48], "lowest": 85, "lsof": 58, "lspci": 58, "lsusb": 58, "ly": 50, "lz": [69, 77], "lzcat": 58, "lzip": [69, 77], "lzma": 58, "lzopcat": 58, "m": [31, 45, 52, 59, 60, 69, 80, 83, 85], "m4": [45, 58], "machin": [8, 9, 14, 21, 22, 40, 43, 45, 48, 50, 63], "machine_arch": [36, 45], "machine_o": 45, "macro": 45, "made": [0, 9, 15, 50, 55, 72, 73, 74, 84, 85], "magic": 48, "mai": [0, 2, 3, 5, 8, 9, 14, 17, 19, 21, 22, 33, 34, 35, 36, 38, 39, 43, 44, 45, 47, 48, 50, 55, 56, 68, 69, 72, 73, 79, 80, 82, 84, 85], "mail": [0, 48, 50], "main": [0, 4, 10, 26, 31, 32, 45, 46, 48, 53, 56, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 84, 85], "mainli": [14, 55], "maintain": [0, 36, 45, 48, 52, 63, 73, 82, 86, 89], "mainten": 48, "major": [0, 3, 17, 22, 26, 36, 48, 50, 73, 85], "make": [0, 1, 9, 17, 19, 31, 34, 41, 42, 43, 44, 45, 48, 52, 54, 55, 56, 58, 59, 60, 63, 64, 65, 73, 79, 80, 82, 83, 85], "makefil": 32, "makemak": 73, "malform": 15, "malfunct": 55, "man": [26, 45, 50, 79], "manag": [0, 10, 11, 17, 19, 26, 45, 53, 55, 65, 69], "mandatori": [17, 21, 22, 45, 55], "mandir": [45, 79], "mangl": 48, "mani": [0, 2, 10, 40, 48, 59, 65, 69, 80, 85], "manifest": [14, 17, 26, 51, 73, 86, 87], "manipul": [53, 65, 84], "manual": [16, 30, 31, 32, 48, 56, 58, 59, 60, 63, 69, 79, 80, 81, 82, 83, 84, 85], "manylinux": 54, "manylinux_x_i": 54, "map": [3, 16, 36], "mapping_at": [11, 16], "mappingnod": [11, 12, 13, 14, 16, 17, 21, 29, 67], "mark": [0, 9, 12, 19, 20, 21, 43, 45, 54, 55, 76, 84], "mark_directori": [11, 18, 19], "mark_download_url": [11, 21], "master": [0, 1, 17, 36, 43, 48, 50, 69, 85], "match": [0, 2, 13, 19, 21, 24, 26, 36, 43, 47, 48, 54, 55, 58, 73, 77, 84], "materi": 56, "matrix": 54, "matter": [0, 45, 56, 62], "max": [16, 43, 79, 84, 85], "max_job": [16, 84], "max_jobs_nod": 16, "maxim": [10, 63], "maximum": [10, 31, 32, 43, 58, 60, 62, 63, 79, 80, 81, 82, 83, 84, 85], "maxjob": [45, 85], "md5sum": 58, "mean": [0, 2, 12, 16, 17, 22, 36, 43, 44, 45, 48, 55, 58, 60, 85], "meaning": [25, 43], "meant": [26, 55, 92], "measur": [0, 4, 82], "mechan": [8, 49, 62, 63], "meet": [65, 87], "megabyt": 85, "member": [0, 16, 45, 56, 87], "memori": [4, 85], "mention": [0, 21, 22, 48, 50, 52, 55, 56, 59, 79], "mere": [31, 59, 73], "merg": 0, "merit": 48, "mesg": 58, "meson": [12, 65, 73], "mess": 48, "messag": [12, 13, 14, 17, 19, 20, 21, 22, 24, 26, 31, 32, 36, 44, 45, 48, 50, 53, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 84, 85, 87], "message_lin": 84, "messeng": 53, "meta": [13, 26, 62, 65], "metacharact": 26, "metadata": [2, 3, 34, 43, 45], "metasourc": 13, "method": [3, 10, 11, 13, 16, 20, 30, 32, 43, 47, 53, 69, 73], "microcom": 58, "microsecond": 85, "middl": [45, 85], "middleproject": 43, "might": [4, 12, 17, 22, 31, 32, 43, 45, 48, 50, 55, 56, 58, 59, 60, 63, 64, 68, 84, 85], "min": [17, 22, 45, 59, 63, 72, 73, 79, 80, 81, 83, 84], "min_vers": 84, "mind": [0, 48, 55, 85, 87], "minim": [0, 1, 45, 54, 85], "minimis": 65, "minimum": [17, 22, 41, 48, 54, 59, 73, 79, 80, 83], "minor": [0, 17, 22, 26, 45], "minut": 85, "mip": 65, "mirror": [11, 13, 17, 21, 29, 31, 32, 36, 43, 44, 50, 58, 60, 62, 63, 67, 71, 79, 80, 81, 82, 83, 84, 89], "mirror_directori": 17, "misbehav": 31, "misc": 34, "miscellan": 11, "misnom": 48, "miss": [15, 26, 56, 58, 69], "missing_fil": [11, 15], "missing_project_conf": [11, 15], "misus": 56, "mix": 48, "mkdir": [1, 42, 69], "mkfifo": 58, "mkpasswd": 58, "mm": 85, "mod_alia": 41, "mod_cgi": 41, "mod_setenv": 41, "mode": [14, 20, 24, 26, 43, 45, 48, 57, 59, 69, 71, 84, 85, 88], "model": [43, 44, 48, 65, 66], "modif": [24, 31, 48, 84], "modifi": [0, 4, 14, 28, 36, 46, 48, 49, 58, 59, 60, 68, 79, 84, 85], "modul": [2, 15, 16, 17, 21, 36, 41, 52, 53, 55, 59, 66, 69, 79, 88], "modular": [48, 73], "modulebuild": 73, "modulenam": 43, "moduleset": [41, 42], "moment": [43, 60, 79], "monitoring_experi": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "more": [0, 2, 5, 9, 14, 16, 17, 21, 22, 24, 25, 26, 30, 31, 34, 36, 39, 40, 43, 44, 45, 48, 50, 52, 55, 56, 62, 63, 67, 68, 69, 73, 80, 82, 84, 85, 86], "morn": 31, "most": [0, 9, 10, 12, 16, 31, 36, 38, 43, 44, 45, 52, 55, 56, 57, 77, 81, 83, 84, 85], "mostli": [16, 31, 44, 48, 64, 79], "mount": [9, 12, 45, 84], "move": [10, 26, 48], "move_atom": [11, 26, 29, 67], "mtime": [11, 23, 24], "much": [0, 4, 43, 48, 50, 52, 60, 62, 80, 81, 83, 85], "mulipl": 43, "multi": 48, "multilin": [14, 17, 21, 22, 85], "multipl": [0, 1, 3, 12, 14, 21, 25, 30, 43, 44, 45, 48, 50, 52, 54, 55, 58, 60, 65, 69, 72, 73, 80, 84, 85], "multiprocess": 10, "musl": 58, "must": [0, 4, 5, 9, 10, 12, 13, 14, 16, 17, 19, 20, 21, 22, 26, 34, 40, 41, 43, 44, 45, 48, 50, 54, 55, 57, 68, 69, 72, 73, 78, 84, 85, 87], "mutat": [14, 39], "mutipl": 43, "my": [45, 69], "my_default": 16, "my_dict": 16, "my_kei": 16, "my_list": 16, "my_map": 16, "my_sequ": 16, "my_valu": 16, "myelement": 84, "mypi": 48, "myproject": [44, 84], "mysourc": [45, 73], "mysourcemirror": 13, "n": [31, 32, 48, 79, 83, 84], "name": [2, 3, 9, 11, 14, 15, 17, 20, 21, 25, 26, 32, 37, 40, 41, 42, 46, 50, 52, 53, 56, 58, 59, 63, 64, 68, 74, 77, 79, 80, 81, 83, 85, 86], "namespac": [9, 14, 17, 45, 48], "nanddump": 58, "nandwrit": 58, "nativ": [0, 45], "natur": [14, 45], "navig": [55, 56], "nbd": 58, "nc": 58, "nearli": 48, "necessari": [8, 31, 39, 48, 69, 85], "necessarili": [0, 9, 82], "need": [0, 1, 2, 3, 5, 9, 10, 12, 13, 16, 17, 19, 20, 21, 22, 26, 30, 31, 32, 36, 40, 41, 42, 43, 44, 45, 49, 50, 51, 52, 54, 55, 56, 57, 58, 59, 61, 62, 63, 64, 65, 68, 69, 72, 73, 79, 80, 82, 83, 84, 85, 86, 87], "needless": 57, "needlessli": 32, "neg": 16, "nelif": 79, "nels": 79, "nest": [17, 19, 44, 45], "netrc": 13, "network": [10, 13, 21, 31, 32, 41, 42, 48, 58, 60, 62, 63, 79, 80, 81, 82, 83, 84, 85, 87], "network_retri": 84, "never": [4, 14, 16, 17, 20, 21, 36, 43, 45, 48, 56, 84, 85], "nevertheless": 63, "new": [4, 11, 12, 13, 14, 15, 16, 17, 20, 22, 26, 31, 32, 40, 44, 45, 48, 50, 52, 55, 56, 57, 59, 63, 64, 72, 73, 74, 80, 84, 85], "newer": [15, 36, 45], "newli": [0, 12, 13, 17, 20, 32, 55, 57, 84], "newlin": [26, 84], "next": [0, 10, 13, 14, 21, 32, 48, 63, 84], "nfi": 79, "nice": [45, 48, 56], "nicknam": 0, "nif": 79, "nightli": 87, "ninja": 85, "nl": 58, "nm": 58, "nmeter": 58, "nocach": 45, "noconfigur": [62, 63, 79], "nocov": 54, "node": [9, 11, 12, 13, 14, 17, 21, 29, 67, 78], "node_at": [11, 16], "node_get_project_path": [11, 17], "node_subst_list": 20, "node_subst_sequence_var": [11, 14], "node_subst_var": [11, 14], "nohup": 58, "nomenclatur": 48, "non": [0, 8, 14, 17, 20, 24, 26, 31, 43, 48, 55, 60, 65, 66, 84, 85], "none": [12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 36, 45, 48, 53, 72, 79, 84, 85], "noprofil": 45, "nor": [48, 62], "norc": 45, "normal": [14, 16, 26, 33, 43, 45, 48, 56, 57, 64, 77, 80, 83, 84, 85], "normal_nam": [11, 12, 14], "notabl": 47, "notat": 48, "note": [0, 2, 4, 5, 13, 17, 21, 31, 32, 34, 36, 39, 43, 44, 45, 48, 53, 55, 56, 58, 60, 62, 63, 66, 73, 77, 80, 84], "noth": [16, 21, 31, 38, 48, 50, 58, 63], "notic": [25, 55, 59, 60, 80], "notifi": 16, "now": [10, 31, 32, 34, 41, 42, 44, 48, 50, 58, 59, 60, 62, 63, 64, 72, 73, 79, 80, 81, 82, 83], "nproc": 58, "nsenter": 58, "nslookup": 58, "ntpd": 58, "null": [9, 59, 79], "number": [0, 16, 17, 22, 43, 45, 48, 54, 55, 68, 73, 84, 85], "nwe": 32, "o": [0, 21, 26, 31, 42, 43, 48, 52, 59, 60, 80, 83, 84, 85], "objcopi": 58, "objdump": 58, "object": [0, 4, 8, 11, 12, 13, 14, 16, 17, 19, 20, 22, 24, 25, 26, 29, 32, 43, 48, 53, 67, 84, 85, 86], "observ": [45, 48, 50, 58, 63, 72, 79, 80, 82, 83, 84, 85], "obstacl": 48, "obtain": [4, 10, 12, 13, 16, 17, 20, 21, 33, 34, 35, 36, 38, 39, 45, 48, 79, 85], "obviou": [48, 56], "occur": [8, 14, 15, 21, 24, 26, 32, 44, 73], "ocean": 55, "od": 58, "odd": 9, "off": [10, 43, 69, 85], "offend": 60, "offer": [0, 10, 43, 48, 59, 63, 65, 73, 80, 85], "offici": 45, "offlin": 85, "often": [5, 48, 58, 84], "oh": 69, "okai": 48, "old": [31, 54, 73], "oldest": 54, "on_error": 84, "onc": [0, 1, 4, 12, 17, 20, 21, 32, 36, 43, 45, 48, 50, 55, 69, 83, 85], "one": [0, 10, 12, 13, 15, 16, 20, 21, 22, 25, 26, 31, 34, 36, 37, 39, 43, 44, 45, 46, 48, 50, 52, 54, 55, 56, 58, 59, 63, 68, 69, 77, 79, 83, 84, 85, 86], "oner": 82, "ones": [12, 48, 82], "ongo": 85, "onli": [0, 2, 3, 5, 9, 10, 12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 25, 26, 31, 32, 33, 34, 35, 36, 37, 39, 40, 43, 44, 45, 48, 50, 52, 55, 56, 60, 63, 65, 69, 80, 83, 84, 85], "onlin": 69, "onto": [10, 34], "onward": 45, "open": [0, 4, 9, 16, 24, 26, 34, 59, 64, 85, 86], "open_directori": [11, 23, 24], "open_fil": [11, 23, 24], "openvt": 58, "oper": [4, 7, 9, 12, 13, 14, 16, 17, 20, 21, 24, 26, 31, 36, 43, 45, 48, 55, 57, 59, 65, 68, 84, 85, 87], "opportun": [44, 56], "oppos": 45, "opt": [12, 44, 63, 79], "optim": [32, 52, 85], "optimi": 85, "option": [1, 4, 9, 13, 14, 17, 19, 21, 22, 24, 26, 28, 30, 31, 32, 33, 34, 35, 38, 39, 40, 43, 44, 48, 53, 55, 56, 59, 65, 69, 71, 72, 75, 76, 77, 83, 84, 92], "orchestr": 85, "order": [0, 3, 5, 6, 9, 10, 12, 14, 17, 20, 21, 22, 25, 26, 31, 32, 36, 43, 44, 45, 46, 50, 52, 53, 55, 56, 57, 58, 60, 63, 64, 68, 80, 81, 82, 84, 85, 86, 87], "org": [33, 34, 35, 38, 39, 41, 42, 45, 50, 52, 55, 62, 63, 65, 73, 79, 85], "organ": [36, 43], "origin": [0, 3, 13, 14, 17, 21, 22, 32, 45, 48, 50, 63, 73, 79, 84, 87], "orphan": [14, 33, 34, 58, 60], "oserror": 26, "osfamili": 87, "ostre": 73, "other": [0, 16, 21, 24, 31, 34, 37, 40, 41, 42, 43, 44, 45, 47, 48, 49, 50, 52, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 65, 66, 67, 68, 69, 79, 83, 84, 85], "otherwis": [8, 12, 14, 16, 20, 24, 25, 36, 48, 63, 82, 84, 85], "our": [0, 5, 17, 22, 32, 45, 50, 54, 55, 56, 58, 59, 62, 63, 65, 79, 80, 81, 82, 83], "ourselv": [60, 83], "out": [8, 17, 31, 32, 34, 36, 41, 42, 45, 48, 50, 55, 56, 58, 59, 63, 68, 69, 73, 81, 82, 84, 85], "outcom": [10, 57, 60], "outlin": [10, 44, 48, 50, 56, 72, 73, 74], "output": [2, 3, 4, 5, 9, 10, 12, 13, 14, 17, 20, 21, 26, 30, 31, 34, 35, 36, 43, 45, 47, 53, 55, 58, 59, 63, 65, 68, 79, 80, 82, 83, 84, 85], "outsid": [4, 9, 15, 17, 19, 48, 66], "outward": [48, 55], "over": [0, 9, 10, 16, 24, 26, 41, 43, 44, 45, 47, 48, 50, 56, 59, 60, 63, 79, 80, 83, 85], "overal": [48, 66, 82], "overlai": 35, "overlap": [11, 14, 25, 28, 36, 45, 58, 71, 90], "overlapact": [11, 14, 25, 29, 67], "overlook": 58, "overrid": [4, 15, 21, 28, 46, 48, 58, 73, 79, 84], "overridden": [12, 21, 36, 43, 45, 48, 58, 79, 85], "overview": [4, 66], "overwrit": [15, 26, 36, 60, 73, 84], "overwritten": [9, 11, 24, 26, 44, 45, 84], "ovserv": 25, "own": [0, 4, 9, 12, 13, 14, 17, 20, 21, 24, 28, 30, 36, 40, 43, 45, 48, 50, 56, 58, 63, 64, 69, 86], "ownership": 0, "oz": [45, 85], "p": [1, 42], "packag": [0, 17, 21, 43, 45, 47, 48, 50, 53, 55, 62, 63, 65, 68, 73, 79], "page": [26, 44, 50, 52, 53, 69, 84, 86, 89], "pair": [16, 19, 84], "paragraph": 56, "parallel": [10, 43, 84, 85], "paramet": [12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 26, 39, 43, 45, 71, 85], "parameteriz": 43, "parent": [12, 20, 26, 34, 36, 44, 45, 48], "pars": [11, 12, 14, 15, 29, 44, 67], "parser": 55, "part": [0, 4, 5, 9, 14, 16, 21, 31, 36, 40, 43, 44, 48, 53, 55, 56, 62], "parti": [0, 17, 43, 48, 53, 68, 82], "partial": [14, 19, 84], "particular": [2, 43, 53, 79], "partit": 85, "partprob": 58, "pass": [0, 10, 12, 13, 14, 17, 21, 26, 31, 34, 45, 48, 50, 53, 55, 69], "passwd": [45, 58], "password": 13, "past": [50, 58, 69], "patch": [3, 36, 48, 50, 56, 58, 69, 73], "path": [0, 1, 4, 11, 12, 14, 15, 17, 19, 20, 21, 23, 26, 31, 34, 36, 44, 48, 55, 58, 59, 62, 69, 73, 75, 78, 79, 80, 81, 83, 84, 85], "pathlik": 17, "pathnam": 24, "pattern": [26, 58, 60, 77], "payload": [12, 13, 17, 20, 21, 45, 87], "pc": 45, "peek": 83, "peer": [45, 85], "peopl": [0, 45, 48], "pep": 45, "pep517": 69, "pep8": 48, "per": [4, 21, 44, 45, 55, 58, 85], "percent": 85, "percentag": 85, "perform": [0, 4, 5, 8, 12, 13, 14, 16, 17, 20, 21, 31, 32, 43, 48, 53, 55, 58, 63, 72, 80, 82, 85, 86], "perhap": [0, 44], "perl": 58, "perl5": 58, "perldoc": 58, "permiss": [20, 33, 34, 35, 38, 39, 45, 48, 85], "permit": [24, 43, 45, 60], "perpetu": 85, "persist": 9, "person": [0, 31, 50, 59], "pertain": 85, "pgrep": 58, "phase": [11, 12, 32, 44, 50], "phoni": [31, 59, 60, 80, 83], "phxqlxaq": 62, "physic": 10, "pick": [48, 57], "pick_first_new": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "pickleabl": 17, "pid": 9, "pin": 54, "pip": [12, 30, 49, 52, 63, 65, 69, 79], "pip3": [49, 58, 69], "pipelin": [4, 11, 12, 13, 15, 17, 20, 31, 32, 33, 36, 40, 43, 45, 47, 50, 52, 58, 60, 62, 63, 65, 73, 79, 80, 81, 82, 83, 84, 85], "pipx": 69, "pkg": [59, 69], "pkg_config_path": 45, "pkgconfig": 45, "pkill": 58, "place": [4, 8, 10, 16, 17, 21, 26, 36, 41, 42, 43, 44, 48, 50, 52, 53, 54, 73, 79, 81, 83, 85], "plai": [4, 45, 48, 63], "plain": [16, 50], "plan": [31, 32, 43, 53, 58, 60, 62, 63, 66, 72, 74, 79, 80, 81, 82, 83], "platform": [11, 15, 24, 43, 45, 48, 54, 65, 66], "pleas": [0, 16, 36, 45, 48, 50, 52, 53, 56, 65, 69, 74], "plot": 43, "plugin": [0, 3, 4, 9, 11, 12, 13, 14, 15, 16, 19, 20, 21, 22, 24, 26, 28, 31, 32, 43, 46, 47, 48, 55, 56, 58, 60, 62, 63, 65, 67, 68, 69, 78, 79, 80, 81, 82, 83, 85], "plugin_conf": [12, 20], "pluginerror": 17, "pmap": 58, "pod2html": 58, "pod2man": 58, "pod2text": 58, "pod2usag": 58, "podcheck": 58, "podselect": 58, "point": [0, 2, 3, 4, 15, 17, 24, 26, 32, 36, 45, 48, 50, 53, 81], "polici": 45, "poll": 8, "pollut": 48, "poni": [13, 17, 82], "pop": 10, "popul": [12, 14, 20, 21, 85], "popular": [65, 85], "port": [41, 42, 61, 84, 85, 86, 87], "portion": [21, 26, 43, 46, 82], "posit": 17, "posix": [8, 24, 26, 45], "possess": 48, "possibl": [0, 2, 9, 10, 20, 25, 26, 31, 33, 34, 36, 39, 40, 43, 44, 45, 46, 48, 50, 55, 56, 60, 63, 84, 85, 86], "possibli": [12, 13, 14, 17, 20, 21, 22, 40, 43, 57, 58, 62, 84], "post": [0, 83], "postfix": 50, "potato": [45, 73], "potenti": [17, 52, 56, 68], "power": [43, 45, 65], "powertop": 58, "practic": [45, 55], "pre": 69, "prebuilt": 45, "precautionari": 82, "preced": [4, 44], "preciou": 50, "precis": 85, "predetermin": 20, "predict": [2, 17, 55, 65], "prefer": [0, 4, 36, 41, 45, 48, 56, 71, 84, 85], "preferr": [36, 45], "prefix": [0, 12, 14, 17, 24, 31, 32, 35, 43, 44, 45, 48, 55, 56, 58, 59, 60, 62, 63, 79, 80, 82, 83, 85], "preflight": [11, 12, 13, 17, 20], "prei": 48, "prepar": [4, 32, 33, 48, 50, 82, 83], "prepend": 43, "preprocessor": 80, "prereleas": 54, "prerequisit": [8, 49], "prescrib": 48, "present": [5, 10, 24, 31, 33, 34, 41, 43, 44, 45, 54, 55, 58, 59, 60, 63, 64, 73, 84, 85, 86], "preserv": [26, 63], "prestag": 45, "pretend": 56, "pretti": [0, 16], "prevent": [10, 48], "previou": [0, 3, 4, 8, 11, 13, 14, 25, 44, 48, 52, 56, 60, 64, 65, 79, 81, 82], "previous": [13, 21, 25, 48, 55, 62, 65], "previous_sources_dir": [13, 21], "prezto": 69, "primari": [0, 21, 55, 59], "primarili": [14, 85], "primit": [9, 26], "principl": 0, "print": [16, 17, 32, 44, 55, 84, 85], "print0": 79, "printabl": 43, "printf": [31, 32, 58, 80, 83], "prioriti": [43, 44, 84, 85, 87], "privat": [0, 16, 45, 85], "privileg": [9, 87], "privilig": 9, "probabl": [0, 56, 63, 64, 79], "problem": [0, 25, 43, 48], "proc": 9, "proce": [12, 85], "procedur": 0, "process": [0, 2, 4, 9, 10, 17, 21, 26, 31, 32, 35, 43, 44, 48, 58, 60, 62, 63, 65, 68, 79, 80, 81, 82, 83, 85], "procf": 9, "produc": [4, 8, 14, 15, 21, 22, 25, 31, 34, 35, 36, 40, 43, 45, 52, 54, 56, 59, 63, 72, 82, 83], "product": [43, 65, 88], "profici": 56, "profil": [1, 11, 15, 53], "prog_not_found": [11, 15], "program": [9, 17, 19, 26, 31, 32, 44, 56, 59, 62, 63, 65, 66], "programnotfounderror": [11, 19, 26, 29, 67], "progress": 53, "prohibit": [9, 45], "proj_path_invalid": [11, 15], "proj_path_invalid_kind": [11, 15], "project": [0, 2, 9, 11, 13, 14, 15, 16, 17, 20, 21, 22, 25, 30, 32, 34, 40, 43, 48, 50, 55, 56, 57, 62, 64, 65, 66, 67, 68, 70, 71, 75, 76, 77, 84, 86, 87, 89, 92], "project_nam": [11, 14, 84], "projectnam": 36, "promis": 45, "prompt": [45, 85], "propag": [34, 36, 48, 59], "proper": [43, 57], "properli": [0, 5, 31, 43, 45, 48, 55, 82, 87], "properti": [4, 24, 43, 46, 48, 55], "propos": [0, 48], "protect": [15, 20, 44], "protected_variable_redefin": [11, 15], "proto": [3, 49], "protobuf": 49, "protoc": 49, "protocol": [0, 3, 8, 41, 86, 87], "prototyp": 48, "proven": [16, 48], "provenanceinform": [11, 16, 29, 67], "provid": [4, 7, 8, 9, 12, 13, 14, 16, 17, 19, 21, 26, 30, 36, 38, 40, 43, 44, 45, 47, 48, 52, 55, 56, 58, 59, 60, 61, 62, 63, 65, 66, 69, 73, 79, 81, 83, 84, 85, 86, 87, 88], "provision": 0, "prune": 85, "ps1": 45, "pscan": 58, "pstat": 52, "pstree": 58, "public": [2, 14, 20, 28, 29, 34, 41, 44, 45, 55, 58, 59, 60, 63, 67, 73, 84, 85], "publicli": 48, "publish": 50, "pull": [2, 5, 8, 10, 40, 45, 68, 85, 86], "puls": 45, "pulse_serv": 45, "pulseaudio": 45, "pure": [2, 55], "purpos": [0, 2, 5, 9, 12, 14, 17, 21, 22, 40, 43, 47, 48, 51, 58, 69, 73], "push": [0, 8, 10, 31, 32, 45, 50, 58, 60, 62, 63, 79, 80, 81, 82, 83, 85, 86, 87], "pusher": [84, 85], "put": [43, 44, 50], "pwd": 40, "pwdx": 58, "py": [0, 8, 17, 44, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57], "py311": 54, "py36": 55, "py37": 55, "pydoc3": 58, "pylint": 55, "pylintrc": 55, "pypa": 54, "pypi": [50, 55, 69], "pytest": 55, "python": [0, 16, 24, 25, 30, 44, 45, 48, 49, 50, 52, 53, 55, 56, 58, 62, 63, 65, 69, 79, 84, 85], "python3": [50, 52, 55, 58, 62, 63, 69, 79], "pythonhost": 73, "pyvenv": 58, "pyx": 52, "qmake": 73, "qualifi": [21, 56], "qualiti": 65, "qualnam": 15, "quantiti": 0, "queri": [3, 10, 13, 21, 31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 84], "question": [12, 13, 20, 21, 36, 43, 48, 65, 85], "queue": [31, 32, 48, 58, 60, 62, 63, 66, 79, 80, 81, 82, 83], "queuestatu": 10, "quicker": 25, "quickli": [0, 43, 48, 85, 92], "quit": [31, 56, 60, 84, 85], "quot": 43, "quota": 85, "r": [24, 55, 79, 84], "rais": [0, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 45], "ran": [55, 83], "random": 9, "rang": [9, 26, 65], "ranlib": 58, "rare": 48, "rate": 85, "rather": [0, 4, 43, 48, 56, 83, 85, 87], "rational": 48, "raw": [1, 48, 56], "rc": 31, "rdate": 58, "rdev": 58, "re": [4, 44, 48, 59, 60, 63, 79, 80, 81, 83, 84], "reach": [0, 48], "read": [4, 9, 12, 13, 16, 17, 19, 20, 21, 26, 39, 40, 43, 45, 47, 48, 69, 79, 82], "readabl": [0, 14, 21, 22, 48, 80, 85], "readahead": 58, "readelf": 58, "reader": [26, 48, 56, 58, 82], "readi": [0, 10, 26, 31, 50, 81, 82, 83], "readili": 2, "readlink": [11, 23, 24, 58], "readm": [50, 52, 69], "readonli": 19, "readprofil": 58, "real": [48, 62, 63, 89], "realli": [25, 56], "realpath": 58, "reapi": [3, 8, 85, 87], "reason": [0, 9, 14, 15, 16, 17, 19, 21, 22, 24, 26, 45, 48, 50, 55, 63, 73, 82, 85], "rebas": [0, 36], "rebuild": [2, 14, 20, 31, 32, 56, 65, 85], "rebuilt": [2, 31, 43, 84, 85], "recal": 32, "recc": 86, "receiv": [17, 36], "recent": [60, 85], "recipi": 50, "recogn": 43, "recommend": [0, 9, 20, 21, 24, 36, 39, 41, 42, 44, 45, 48, 55, 56, 65, 69, 73, 74, 84, 86], "recompil": 52, "reconcil": [36, 45], "reconfigur": 32, "record": 85, "recover": 48, "recreat": [32, 55], "recurs": [5, 14, 15, 16, 24, 44, 45, 63], "recursive_includ": [11, 15], "red": 63, "redeclar": 45, "redefin": [36, 48, 63], "redesign": [48, 73, 74], "reduc": [2, 56, 85], "redund": [48, 50, 56, 73, 85], "redundantli": [36, 43, 45, 48], "ref": [2, 10, 11, 12, 13, 15, 17, 20, 25, 28, 36, 43, 44, 48, 53, 56, 58, 63, 67, 73, 76, 77, 79, 82, 83, 84, 85, 87], "ref_not_in_track": [11, 25], "refactor": 55, "refer": [1, 3, 4, 10, 11, 13, 15, 25, 36, 37, 44, 45, 46, 48, 55, 56, 58, 61, 62, 63, 64, 65, 68, 74, 84, 85, 87], "referenc": [0, 15, 17, 43, 45, 62, 85], "reflect": [13, 21, 50], "refresh": 17, "regard": [4, 12, 14, 34, 48, 73], "regardless": [0, 4, 24, 31, 45, 72, 85], "regener": 50, "regist": 52, "register_cython_modul": 52, "registri": 87, "regress": [0, 48, 52, 55], "regular": [0, 9, 17, 24, 43, 45, 48, 62], "regular_fil": [11, 23, 24], "regularli": 63, "reimplement": 25, "rel": [12, 14, 17, 19, 20, 24, 26, 34, 36, 43, 44, 45, 48, 56, 75, 76, 84, 85], "relat": [0, 5, 15, 47, 48, 50, 58, 73, 85, 90], "relationship": [5, 12, 14, 31, 43, 44, 45, 48, 68], "relav": 3, "releas": [0, 26, 36, 43, 45, 48, 69, 73, 77, 79], "release_text": 43, "relev": [3, 22, 36, 43, 45, 52, 85], "reli": [0, 36, 55, 86], "reliabl": [63, 65, 73], "relink": 31, "reload": 85, "reloc": [44, 63], "remain": [10, 22, 48, 56, 73, 74, 80, 83, 85], "remaind": 50, "rememb": [21, 44, 55, 69], "remot": [2, 3, 4, 9, 10, 11, 13, 15, 17, 21, 30, 31, 32, 40, 43, 45, 58, 59, 60, 62, 63, 64, 65, 66, 68, 69, 71, 72, 79, 80, 81, 82, 83], "remov": [0, 11, 14, 16, 23, 24, 26, 32, 45, 50, 58, 60, 72, 73, 74, 79, 82, 84, 85], "renam": [11, 23, 24, 26, 48], "render": [48, 53], "renic": 58, "reopen": 32, "repeat": 3, "replac": [2, 14, 17, 24, 26, 41, 42, 44, 85], "repo": [21, 41, 57], "report": [10, 13, 14, 17, 19, 21, 22, 24, 26, 48, 53, 55, 81], "report_written": 26, "repositori": [0, 32, 36, 41, 43, 45, 49, 50, 52, 68, 69, 73, 81, 83], "repres": [10, 13, 14, 16, 21, 24, 25, 40, 44, 53, 54, 62, 68, 78, 85], "represent": [4, 40], "reproduc": [2, 4, 9, 45, 65, 83], "repurpos": 48, "request": [10, 13, 17, 21, 84, 85, 87], "requir": [2, 3, 5, 8, 9, 10, 12, 13, 14, 15, 17, 19, 20, 21, 22, 26, 32, 33, 34, 35, 36, 38, 39, 41, 43, 45, 48, 49, 52, 54, 55, 56, 58, 59, 60, 62, 63, 65, 68, 71, 72, 73, 79, 80, 81, 82, 83, 84, 85, 86], "rerais": 26, "rerun": [32, 79], "resembl": 80, "reserv": [43, 85], "reset": [12, 32, 34, 58, 69], "resid": [43, 47, 48, 85], "resiz": 58, "resolut": [9, 43], "resolv": [4, 9, 13, 14, 15, 21, 31, 32, 36, 37, 44, 45, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 85], "resort": [16, 26], "resourc": [0, 8, 9, 21, 66, 85], "respect": [0, 20, 21, 44, 48, 63, 79, 85], "respond": [0, 84], "respons": [0, 4, 8, 10, 21, 48], "rest": 0, "restart": [69, 86], "restrict": [9, 45, 68, 73], "restructuredtext": 56, "result": [1, 2, 3, 4, 8, 9, 10, 12, 14, 16, 21, 24, 25, 26, 31, 36, 43, 45, 46, 48, 50, 51, 52, 56, 57, 58, 59, 65, 80, 83, 84, 85, 86], "resum": 10, "retain": [48, 85], "retri": [8, 31, 32, 48, 58, 60, 62, 63, 79, 80, 81, 82, 83, 84, 85], "retriev": [2, 8, 13, 16, 21, 48, 65], "return": [8, 12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 25, 26, 32, 48, 59, 80, 83], "reus": [2, 31, 32, 36, 56, 85], "revers": [2, 4, 10, 16, 31, 34, 36, 40, 45, 59, 60, 82], "revert": 59, "review": [0, 48], "revis": [17, 22, 45, 56, 62], "rewrit": 55, "rewritten": 55, "rfkill": 58, "richer": 52, "rid": 60, "right": [0, 2, 15, 36, 50, 55, 63], "risk": 58, "rivet": 55, "rm": [60, 69, 79], "robust": [0, 55], "role": [4, 45, 56, 63], "roll": 50, "root": [3, 9, 12, 14, 19, 20, 21, 24, 31, 34, 35, 39, 41, 42, 43, 44, 45, 58, 59, 60, 73, 77, 79, 80, 81, 83, 85], "root_read_onli": [19, 20], "rough": 10, "row": 80, "rst": [50, 56, 57, 69], "rstrip": 48, "rule": [0, 5, 12, 13, 17, 20, 28, 33, 34, 44, 48, 59], "run": [0, 3, 4, 5, 8, 9, 10, 11, 13, 14, 16, 17, 18, 19, 20, 21, 26, 27, 30, 32, 33, 36, 43, 45, 46, 47, 48, 50, 51, 52, 53, 54, 56, 57, 60, 64, 69, 71, 81, 84, 85, 87, 89, 92], "run_cleanup_command": [11, 14], "runner": 0, "runtim": [2, 5, 12, 14, 19, 20, 31, 34, 40, 45, 47, 51, 54, 59, 60, 63, 65, 73, 82, 83, 84, 85, 87], "rust": 65, "safe": [10, 14, 16, 50], "safe_copi": [11, 26, 29, 67], "safe_del": [11, 16], "safe_link": [11, 26, 29, 67], "safe_remov": [11, 26, 29, 67], "safeti": 9, "sai": [0, 4, 12, 13, 17, 20, 32, 43, 45, 46, 48, 57, 59, 79, 85], "said": [34, 45, 48, 56], "sail": 55, "sake": [31, 48, 58, 60, 62, 83], "same": [0, 1, 2, 3, 4, 8, 10, 12, 13, 14, 17, 21, 22, 24, 25, 26, 31, 34, 36, 40, 43, 44, 45, 46, 48, 50, 52, 55, 56, 57, 58, 60, 63, 64, 68, 69, 72, 74, 79, 80, 81, 82, 83, 85], "sampl": [17, 31, 56, 59, 60, 80, 83], "sandbox": [4, 11, 12, 14, 15, 20, 24, 25, 29, 30, 31, 32, 35, 39, 48, 55, 58, 59, 60, 62, 63, 65, 66, 67, 69, 79, 80, 81, 82, 83, 84, 87], "sandboxcommanderror": [11, 18, 19, 29, 67], "sandboxerror": 19, "sandboxremot": 8, "sane": [43, 85], "satisfi": 64, "save": [0, 4, 11, 26, 48, 72], "save_config": 48, "save_file_atom": [11, 26, 29, 67], "sbin": [45, 58, 62, 63, 79], "sbindir": [45, 47, 62, 63, 79], "scalar": 16, "scalar_at": [11, 16], "scalarnod": [11, 14, 16, 17, 29, 67], "scanelf": 58, "scenario": [5, 48, 59, 85], "schedul": [17, 48, 66, 71, 72], "scheme": [83, 85], "scope": [14, 66, 85], "scp": 50, "scratch": [65, 81], "script": [0, 1, 11, 12, 14, 29, 30, 32, 50, 57, 62, 67, 69, 72, 84], "scriptel": [11, 29, 67], "sdist": [50, 51], "sdk": [12, 35, 62, 65], "search": [0, 11, 14, 26, 56, 84], "seccomp": 9, "second": 85, "secondari": 55, "secondli": 4, "section": [4, 8, 9, 12, 28, 30, 31, 32, 43, 44, 45, 47, 48, 50, 52, 54, 56, 61, 64, 66, 67, 68, 69, 71, 73, 79, 80, 81, 85, 88, 90, 91], "secur": [36, 41, 42, 85], "see": [0, 2, 8, 9, 13, 17, 19, 20, 21, 25, 30, 31, 32, 33, 34, 35, 38, 39, 41, 42, 43, 44, 45, 48, 50, 52, 55, 56, 57, 58, 59, 60, 62, 63, 68, 69, 75, 76, 77, 80, 82, 83, 85, 86], "seek": 48, "seem": 9, "seen": [31, 48, 82], "segment": 48, "select": [5, 14, 33, 43, 45, 63, 69], "selector": 5, "self": [12, 13, 14, 17, 21, 22, 48, 77], "semant": [45, 48], "send": [8, 50], "sendmail": 58, "sens": [24, 48, 59, 62, 64], "sensibl": [48, 58], "sent": [0, 85], "sentenc": [50, 56], "sentinel": 16, "separ": [0, 9, 11, 14, 24, 26, 30, 31, 32, 43, 45, 46, 47, 48, 52, 53, 54, 55, 56, 59, 62, 63, 72, 73, 80, 84, 85], "seper": 85, "seq": 58, "sequenc": [14, 16, 17], "sequence_at": [11, 16], "sequencenod": [11, 14, 16, 29, 67], "seri": [0, 56, 57], "serial": [21, 49], "serv": [9, 34], "server": [3, 8, 10, 17, 40, 43, 49, 50, 68, 71, 73, 84], "servic": [5, 8, 42, 54, 71, 86, 87], "session": [2, 10, 21, 31, 32, 50, 58, 60, 62, 63, 79, 80, 81, 82, 83, 84, 85], "set": [4, 9, 12, 13, 14, 15, 17, 19, 20, 21, 22, 24, 25, 31, 36, 39, 43, 44, 45, 47, 48, 52, 53, 54, 55, 56, 58, 63, 65, 69, 71, 77, 79, 84, 85], "set_": 48, "set_count": 48, "set_environ": [11, 18, 19], "set_foo": 48, "set_install_root": [11, 20], "set_public_data": [11, 14], "set_ref": [11, 13, 21], "set_root_read_onli": [11, 20], "set_supported_alias": [11, 22], "set_work_dir": [11, 20], "set_work_directori": [11, 18, 19], "setenv": 41, "setfont": 58, "setgid": 9, "setkeycod": 58, "setlogcon": 58, "setsid": 58, "setuid": 9, "setup": [11, 17, 20, 29, 43, 45, 49, 50, 51, 52, 56, 67, 68, 79, 85], "setuptool": [12, 17, 50, 55, 69, 73], "sever": [9, 32, 45, 48, 58], "sh": [12, 14, 31, 45, 59, 62, 63, 79, 82, 87], "sha1sum": 58, "sha256": [2, 3, 12, 13, 17, 20, 26], "sha256sum": [11, 26, 29, 58, 67, 76, 77], "sha3sum": 58, "sha512sum": 58, "shard": [84, 85], "share": [3, 4, 8, 31, 36, 45, 52, 58, 59, 60, 62, 63, 68, 79, 80, 82, 85], "sharedstatedir": [45, 62, 63, 79], "shed": 0, "shell": [5, 8, 9, 12, 14, 28, 31, 32, 43, 50, 56, 58, 59, 62, 63, 69, 79, 80, 82, 83, 85], "ship": 55, "short": 48, "shortcut": 65, "shorthand": [43, 45, 56], "shortlog": 50, "should": [0, 1, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 21, 22, 25, 26, 32, 33, 34, 36, 40, 43, 44, 45, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 59, 60, 69, 72, 73, 76, 82, 83, 84, 85], "shouldnotlink": 79, "show": [0, 4, 7, 16, 31, 36, 63, 79, 80, 81, 83, 85], "show_default": 53, "showkei": 58, "shown": [4, 17, 45, 48, 58, 84], "shred": 58, "shuf": 58, "shutil": 26, "side": [0, 4, 31, 36, 44, 45, 48, 86, 87], "sign": [50, 85], "signal": [0, 8, 21], "signatur": 48, "signific": [12, 20, 73], "significanli": 36, "significantli": [36, 73], "silenc": [17, 60], "silent": 9, "silent_nest": 17, "similar": [2, 4, 31, 45, 48, 55, 72, 79, 82, 85], "similarli": [8, 36, 53, 55, 85, 86], "simpl": [5, 25, 32, 34, 43, 45, 46, 48, 57, 59, 80, 81, 83, 84, 85, 86, 87], "simple_task": 53, "simpler": [3, 4, 44], "simpli": [5, 12, 13, 17, 20, 35, 36, 40, 44, 45, 50, 55, 56, 62, 63, 79, 80, 81, 82, 84], "simplif": 10, "simplifi": 43, "simultan": [84, 85], "sinc": [2, 9, 10, 13, 21, 22, 33, 43, 45, 48, 50, 55, 60, 62, 77, 80, 84, 85, 86], "singl": [21, 24, 25, 26, 36, 43, 44, 45, 46, 48, 56, 58, 59, 73, 84, 85], "sink": 55, "sit": 50, "site": [55, 62, 63, 79], "sitepackag": 55, "situat": [21, 45, 60], "size": [0, 11, 23, 24, 58, 85], "skip": [10, 31, 32, 54, 55, 58, 59, 60, 62, 63, 79, 80, 81, 82, 83, 84, 85], "slash": [0, 43, 84], "sleep": 87, "slightli": [2, 56, 79], "slow": [25, 55], "small": [0, 48, 85], "smemcap": 58, "snakeviz": 52, "snapshot": [50, 56, 69], "snippet": [43, 86, 87], "so": [0, 1, 9, 10, 12, 13, 14, 16, 17, 20, 21, 31, 32, 36, 39, 43, 45, 47, 48, 50, 54, 55, 56, 58, 59, 60, 73, 79, 80, 81, 83, 84, 85, 86, 89], "sock": 87, "socket": [15, 17, 26, 45], "soft": [12, 32, 84], "softwar": [14, 33, 34, 35, 36, 38, 39, 40, 43, 45, 50, 61, 63, 65, 82, 83, 85], "sole": 34, "solut": [43, 60, 65], "solv": 73, "somber": 80, "some": [0, 1, 5, 7, 9, 10, 12, 13, 14, 15, 17, 21, 22, 24, 25, 26, 31, 32, 35, 36, 39, 40, 43, 44, 45, 47, 48, 49, 50, 55, 56, 57, 58, 59, 63, 65, 68, 69, 79, 80, 82, 83, 84, 85, 86, 87], "somefil": 75, "someon": 0, "someth": [0, 12, 13, 16, 17, 20, 26, 33, 41, 42, 48, 56], "sometim": [10, 36, 45, 48, 64, 82, 85], "somewher": [21, 83], "sophia": 59, "sort": [10, 12, 20, 58, 60], "sourc": [0, 2, 8, 9, 10, 11, 14, 15, 17, 20, 24, 25, 26, 29, 31, 32, 34, 41, 42, 44, 46, 48, 50, 51, 53, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 71, 73, 75, 76, 77, 78, 79, 80, 82, 83, 86, 87], "source_date_epoch": 45, "source_remot": 84, "source_url": [13, 22], "sourcedir": 85, "sourceerror": [11, 12, 13, 17, 20, 21, 24, 26, 29, 67], "sourcefetch": [11, 29, 67], "sourcemirror": [11, 21, 29, 67], "sourcemirrorerror": [11, 22, 29, 67], "sourceref": [11, 13, 21, 25, 29, 67], "space": [43, 56, 85], "sparc": 45, "speak": [0, 4], "special": [9, 31, 45, 48, 56, 58, 59, 63, 68, 73, 80, 85], "specif": [4, 8, 9, 10, 12, 13, 21, 31, 33, 34, 35, 36, 38, 39, 43, 44, 45, 48, 50, 55, 58, 63, 65, 67, 71, 73, 82, 83, 84], "specifi": [4, 9, 12, 13, 14, 15, 16, 17, 20, 21, 22, 24, 25, 26, 30, 33, 34, 35, 36, 37, 39, 40, 43, 44, 45, 47, 48, 53, 54, 55, 56, 58, 59, 63, 69, 72, 75, 76, 77, 78, 79, 80, 81, 83, 84, 85, 87], "speed": 52, "spefi": 84, "spent": 52, "sphinx": [48, 56], "spin": [86, 87], "split": [5, 14, 28, 33, 34, 44, 48, 56, 59, 60, 73, 85], "spoken": [14, 33, 34], "src": [14, 24, 26, 43, 44, 49, 52, 57, 69], "ss": 85, "ssh": 41, "ssl": [41, 42], "ssl_client": 58, "stabil": [12, 13, 17, 20, 48, 63], "stabl": [45, 48, 50, 54, 69], "stack": [6, 30, 31, 32, 44, 58, 60, 61, 62, 63, 65, 79, 80, 82, 83, 85], "stage": [2, 3, 4, 8, 9, 11, 13, 14, 17, 19, 20, 21, 25, 30, 31, 32, 33, 35, 36, 39, 43, 45, 47, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 84], "stage_artifact": [11, 14, 25, 47], "stage_dependency_artifact": [11, 14, 25], "stage_directori": [11, 21], "stage_sourc": [11, 12, 14], "stai": 56, "stale": 0, "stand": 48, "standalon": 56, "standard": [9, 17, 48, 55, 63, 69, 77, 84, 87], "starch": 45, "start": [0, 1, 2, 10, 14, 15, 17, 19, 31, 32, 45, 48, 50, 52, 56, 58, 59, 60, 62, 63, 64, 69, 71, 79, 80, 81, 82, 83, 85, 87], "startup": [0, 85], "starv": 17, "starvat": 10, "stat": [11, 23, 24], "state": [2, 4, 11, 13, 14, 32, 43, 50, 53, 72, 81, 84, 85], "statement": [17, 22, 34, 44, 45, 48, 59, 60, 63, 80], "static": [43, 45, 54, 56, 69], "statu": [10, 11, 17, 24, 31, 32, 53, 58, 59, 62, 63, 64, 79, 80, 82, 83, 85], "stderr": [8, 17, 31, 32, 55, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 85], "stdin": 17, "stdio": [32, 59, 80, 83], "stdout": [8, 17, 55, 84, 85], "step": [32, 54, 55, 56, 58, 63, 65, 81, 84, 89, 92], "still": [0, 2, 5, 9, 31, 45, 48, 50, 54, 55, 63, 85], "stop": [69, 86], "storag": [11, 24, 46, 66, 69, 84, 85, 86, 87], "store": [2, 3, 4, 8, 9, 12, 13, 16, 17, 21, 26, 34, 43, 45, 46, 48, 55, 56, 59, 63, 68, 79, 80, 81, 83, 84, 85, 86], "str": [12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 25, 26], "stranger": [31, 59], "strate": 85, "strategi": [36, 60], "stream": [11, 15, 17, 53], "streamlin": 65, "strength": 55, "strict": [14, 20, 32, 43, 45, 58, 60, 62, 63, 66, 71, 79, 80, 81, 82, 83, 84, 88], "strictli": 4, "string": [12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 26, 43, 44, 45, 47, 48, 53, 56, 58, 77, 80, 84, 85], "strip": [11, 38, 45, 48, 58, 79, 83], "strip_node_info": [11, 16], "strive": 48, "strong": [2, 3, 73], "strongli": [0, 20, 41, 42, 73], "structur": [3, 9, 11, 14, 16, 28, 45, 66, 81], "stub": 44, "stuff": 26, "style": [9, 43, 47, 56, 84, 85], "sub": [5, 43, 44, 45, 53], "subclass": [10, 16, 17, 48], "subcommand": [53, 71], "subdir": [12, 63, 79], "subdirectori": [4, 12, 20, 21, 31, 32, 43, 45, 55, 56, 58, 59, 60, 62, 63, 64, 77, 79, 80, 81, 82, 83, 84, 85], "subdirectory_spec": 24, "subject": 56, "submiss": 0, "submit": [36, 56], "submitt": 0, "submodul": [17, 21, 45], "subpath": 36, "subprocess": [10, 17], "subproject": [14, 15, 30, 37, 43, 44, 62, 68, 71, 73, 85, 91], "subproject_inconsist": [11, 15], "subsequ": [10, 12, 19, 32, 44], "subset": [14, 25, 30, 35, 43, 59, 69], "substitut": [14, 15, 20, 21, 47, 79], "subsubproject": 36, "subsystem": 40, "subtyp": 16, "succe": 12, "success": [8, 14, 17, 31, 32, 40, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 85], "successfulli": [0, 8, 48, 50, 85], "suffer": [25, 85], "suffici": [10, 43, 55], "suffix": [17, 21, 85], "suggest": [0, 79, 82, 84], "suit": [0, 40, 48, 54], "suitabl": [0, 2, 14, 17, 41, 42, 43, 86], "sum": [12, 13, 17, 20, 58], "summar": [50, 63], "summari": [0, 12, 32, 48, 55, 62, 85], "suno": 45, "super": 48, "superfici": 50, "suppli": [1, 43, 45], "support": [0, 9, 12, 13, 14, 15, 17, 21, 22, 36, 43, 44, 45, 46, 48, 53, 55, 58, 60, 63, 65, 68, 69, 72, 73, 80, 83, 84, 85, 87], "supportedcompress": 16, "suppos": 32, "suppress": 17, "sure": [9, 19, 44, 45, 48, 50, 54], "surfac": [43, 55, 63], "surpris": 48, "surround": 48, "suspend": [10, 85], "sv": 69, "switch": [2, 84], "symbol": [3, 12, 13, 15, 17, 21, 22, 24, 26, 30, 36, 37, 43, 45, 46, 52, 55, 73, 80, 83, 84, 86], "symlink": [11, 23, 24], "symptom": 26, "syntax": [43, 44, 45, 64, 83], "synthet": 78, "syscal": [9, 17], "sysconfdir": [43, 45, 47, 62, 63, 79], "sysroot": [14, 65, 72, 83, 84], "system": [2, 5, 8, 12, 14, 21, 24, 26, 39, 40, 43, 45, 47, 48, 52, 54, 55, 59, 63, 65, 69, 73, 80, 82, 83, 85, 87], "t": [0, 1, 2, 14, 16, 21, 24, 25, 41, 42, 43, 45, 48, 50, 55, 56, 58, 66, 68, 69, 79, 80, 84, 85], "t1": 17, "tab": 69, "tabl": 69, "tac": 58, "tag": [13, 21, 50, 54, 69], "tail": 58, "take": [0, 4, 8, 9, 12, 14, 17, 21, 25, 31, 44, 48, 50, 52, 55, 59, 62, 63, 79, 80, 81, 82, 83, 84, 85], "taken": [32, 43, 45, 48, 50, 79, 85], "talk": 10, "tangenti": 56, "tar": [24, 30, 31, 32, 50, 58, 60, 62, 63, 69, 71, 72, 73, 79, 80, 82, 83, 84, 89], "tarbal": [12, 13, 17, 20, 43, 45, 50, 51, 59, 69, 73, 77, 79, 80, 83, 84], "tarfil": 24, "target": [12, 15, 17, 26, 28, 31, 32, 35, 36, 37, 40, 43, 44, 55, 56, 57, 58, 60, 62, 63, 65, 72, 79, 80, 81, 82, 83, 84, 85, 86, 87], "target_directori": 84, "task": [10, 13, 17, 21, 24, 31, 32, 48, 53, 56, 58, 60, 62, 63, 79, 80, 81, 82, 83, 84, 85], "tax": 82, "tcc": 58, "team": 65, "technic": [0, 9, 48], "techniqu": [48, 65, 68], "technologi": 65, "tee": 58, "tell": [17, 32], "tempdir": [11, 21, 26], "tempfil": 21, "temporari": [14, 17, 21, 22, 26, 48, 84], "tempt": 48, "tend": 56, "term": [43, 45, 48, 55], "termin": [10, 21, 26, 84, 85], "terminologi": 56, "ters": 56, "test": [0, 14, 15, 21, 22, 31, 32, 44, 45, 48, 50, 51, 52, 54, 56, 58, 59, 62, 63, 79, 80, 82, 83, 85, 86], "test_build_track": 55, "test_wheel": 54, "testsuit": 48, "testutil": 57, "text": [24, 31, 43, 48, 50, 53, 56, 85], "tgz": [13, 43, 72], "than": [3, 4, 5, 13, 15, 16, 21, 25, 36, 41, 42, 43, 45, 48, 52, 56, 68, 82, 85, 87], "thei": [0, 3, 6, 9, 10, 12, 14, 16, 17, 20, 21, 24, 26, 31, 33, 34, 36, 40, 43, 44, 45, 46, 48, 50, 55, 56, 58, 60, 61, 62, 63, 68, 69, 72, 73, 82, 83, 84, 85, 89], "them": [0, 5, 8, 9, 12, 13, 17, 20, 21, 32, 36, 43, 44, 46, 48, 50, 55, 56, 58, 68, 82, 84, 85, 86], "themselv": [56, 63], "therefor": [9, 16, 55], "therein": [14, 45], "thi": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 63, 64, 66, 67, 68, 69, 70, 71, 72, 73, 74, 77, 78, 79, 80, 81, 82, 83, 84, 85, 87, 88, 89, 90, 91, 92], "thing": [0, 9, 21, 45, 48, 50, 55, 57, 58, 63, 65], "think": [48, 81], "third": [8, 17, 43, 48, 53, 68, 82], "those": [0, 12, 14, 16, 26, 41, 43, 48, 52, 55, 79, 82, 83, 85], "though": [0, 48], "thousand": [58, 65], "thread": 85, "three": [43, 80, 85], "throttl": 85, "through": [0, 9, 10, 13, 17, 30, 34, 43, 45, 46, 48, 53, 55, 56, 73, 86], "throughout": [0, 44, 46, 56], "throw": [16, 26], "thrown": [16, 26, 45], "thu": [2, 34, 45], "thumb": [12, 13, 17, 20, 48], "time": [0, 2, 4, 5, 10, 12, 13, 14, 17, 21, 24, 25, 30, 31, 32, 40, 43, 44, 45, 48, 50, 51, 52, 53, 55, 56, 57, 58, 59, 60, 63, 65, 68, 72, 79, 80, 82, 83, 84, 85], "timecod": 85, "timed_act": [11, 17, 53], "timefram": 0, "timeout": [17, 48, 58, 87], "timestamp": [2, 83], "tip": [0, 56], "titl": [0, 50, 56], "tmp": [9, 45], "tmpcafebeef": 26, "toctre": 56, "togeth": [36, 43, 47, 80, 83], "token": [10, 85], "tomjon": 45, "too": [9, 36, 48, 53, 55, 56, 83], "took": 85, "tool": [4, 9, 12, 13, 17, 19, 20, 26, 36, 39, 43, 45, 49, 54, 55, 58, 63, 65, 69, 83, 86], "toolchain": [36, 65, 69], "toolnam": 17, "top": [4, 9, 19, 35, 36, 44, 45, 48, 50, 58, 71, 72], "topic": [50, 52, 56, 82], "toplevel": [36, 43, 45, 46, 48, 55, 56, 73, 80, 85], "total": [31, 32, 58, 60, 62, 63, 79, 80, 81, 82, 83, 85], "touch": [79, 81], "tox": [48, 55, 56, 62, 63, 79], "tpwfzpoh": 82, "tr": 58, "trace": [48, 85], "trace_record_callop": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "traceabl": [48, 65], "tracerout": 58, "traceroute6": 58, "track": [0, 2, 4, 10, 11, 13, 17, 25, 34, 36, 43, 45, 46, 48, 62, 65, 71, 76, 77], "traffic": 85, "trail": [48, 72, 84], "trailing_list_direct": [11, 15], "transfer": 85, "transform": [16, 65, 68], "transient": [14, 31, 58, 59], "transit": [5, 54, 73], "translat": [21, 22, 45, 79], "translate_url": [11, 13, 21, 22], "transmit": 8, "transpar": 34, "travers": 14, "treat": [9, 16, 32, 43, 45, 50, 83, 87], "tree": [9, 14, 24, 58, 72, 81, 83, 84, 85], "tri": [10, 63], "trigger": [16, 44, 45, 50, 85], "trim": [55, 83], "tripl": 84, "trivial": [0, 48], "true": [12, 13, 14, 16, 17, 20, 21, 24, 26, 31, 33, 34, 36, 44, 45, 48, 53, 56, 58, 59, 62, 63, 76, 79, 82, 84, 85, 86, 87], "truncat": 58, "trust": [14, 89], "truthi": [24, 44, 45], "try": [15, 26, 36, 45, 48, 50, 52, 60, 84, 85, 86], "tty": 58, "ttysiz": 58, "tupl": [13, 17, 26], "turn": [43, 52, 62, 63, 79, 85], "tutori": [32, 41, 56, 65, 71], "tweak": 43, "twice": 45, "two": [2, 4, 5, 8, 9, 21, 24, 31, 45, 48, 56, 57, 59, 63, 68, 81, 84, 85, 86], "txt": [54, 55, 59, 60, 75], "type": [4, 5, 8, 10, 11, 14, 15, 17, 21, 24, 29, 33, 36, 44, 45, 48, 53, 59, 63, 66, 67, 68, 80, 83, 84, 85, 86, 87], "typeerror": 16, "typic": [8, 12, 21, 31, 32, 35, 44, 47, 55, 60, 83, 86, 87], "typo": 16, "tz": 45, "u": [0, 30, 32, 45, 50, 53, 56, 59, 80, 81, 85], "ubuntu": 69, "udhcpc6": 58, "ui": [0, 50, 85], "uid": [9, 43, 45], "uk": [41, 42], "ultim": [21, 80], "unabl": 50, "unaccept": 48, "unalias": [25, 45], "unaliased_url": [11, 25], "unam": 45, "unambigu": 0, "unauthent": [86, 87], "unavail": 48, "uncertainti": 0, "unchang": [48, 74], "uncommit": 50, "uncommon": 21, "uncondition": 32, "undefin": 15, "under": [9, 20, 32, 33, 34, 35, 38, 39, 43, 45, 48, 52, 55, 57, 62, 83, 85, 89], "underli": [15, 16, 17, 31, 44, 60, 69], "underscor": [45, 48], "understand": [4, 5, 10, 19, 34, 48, 56, 57, 58, 83], "understood": [4, 38, 43, 46, 47, 82, 83, 84], "unexpand": 58, "unexpect": [26, 48], "unexpectedli": 57, "unfilt": 59, "unfinish": 50, "unfortun": [48, 56, 87], "uniform": 48, "unilater": 48, "unimpl": 21, "uniq": 58, "uniqu": [3, 4, 11, 12, 13, 14, 17, 20, 24, 40, 45, 59, 65, 73, 79, 80, 81, 83], "unique_id": 13, "unit": [4, 55], "unix": [9, 26, 45, 87], "unix2do": 58, "unless": [12, 14, 21, 22, 26, 33, 34, 35, 38, 39, 45, 48, 58, 84, 85, 86], "unlik": [4, 9, 40, 45, 48, 82], "unlink": [26, 58], "unlzma": 58, "unlzop": 58, "unnam": [86, 87], "unnecessari": [0, 12, 14, 85], "unneed": 26, "unpack": [4, 43], "unprivileg": 9, "unquantifi": 0, "unrecogn": [16, 55], "unrel": [55, 56], "unresolved_vari": [11, 15], "unset": 85, "unshar": 58, "unspecifi": [45, 84, 85], "unstabl": [45, 50], "unstag": 25, "unstaged_fil": [11, 25], "unsuit": [12, 13, 17, 20], "unsupport": [12, 13, 17], "unsupported_plugin": [11, 15], "unsupported_project": [11, 15], "unsur": 55, "unsuspect": 45, "until": [0, 8, 10, 12, 19, 26, 44, 45, 60, 85], "untyp": 16, "unus": 85, "unwant": [59, 60], "unxz": 58, "unzip": 58, "up": [0, 9, 12, 20, 21, 22, 31, 32, 43, 45, 48, 50, 52, 56, 57, 64, 69, 71, 80, 84, 85, 87], "updat": [0, 2, 13, 14, 21, 31, 36, 40, 43, 46, 47, 50, 51, 55, 58, 60, 76, 77, 84, 85], "upgrad": 63, "upload": [40, 43, 50, 84, 85], "upon": [56, 68, 85], "upstream": [0, 4, 36, 43, 48, 50, 54, 63, 76, 77, 79, 86], "uptim": 58, "urandom": 9, "uri": [11, 21, 29, 43, 45, 67, 84, 85], "url": [11, 13, 17, 22, 25, 29, 30, 36, 41, 42, 43, 45, 50, 58, 63, 67, 73, 77, 79, 82, 83, 84, 85, 86, 87], "us": [0, 1, 2, 3, 4, 5, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 28, 29, 30, 32, 33, 34, 35, 36, 37, 38, 39, 44, 45, 46, 47, 48, 49, 50, 52, 53, 54, 56, 61, 62, 64, 66, 68, 69, 70, 72, 74, 76, 77, 78, 81, 85, 86, 87, 89, 92], "usag": [16, 31, 70, 75, 76, 77, 78, 80, 84, 86, 87], "use_filt": 59, "user": [0, 3, 4, 10, 12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 31, 32, 36, 38, 40, 43, 44, 45, 48, 49, 50, 53, 55, 58, 59, 60, 62, 63, 64, 65, 68, 69, 71, 74, 79, 80, 81, 82, 83, 84, 86, 87, 88, 92], "user_assert": [11, 15], "userchroot": 9, "usernam": [0, 45], "using_exampl": 56, "using_tutori": 56, "usr": [31, 32, 41, 43, 44, 45, 47, 58, 59, 60, 62, 63, 69, 79, 80, 82, 83], "usual": [0, 4, 12, 13, 16, 17, 20, 21, 45, 48, 50, 57, 58, 83, 84, 85], "utc": 45, "utf": [2, 24, 48], "util": [4, 11, 12, 13, 15, 17, 20, 29, 47, 48, 52, 67, 73], "utilerror": [11, 26, 29, 67], "utim": 48, "utmost": 48, "uudecod": 58, "uuencod": 58, "uz": 69, "v": [31, 87], "v1": [32, 58, 62, 63, 79, 80, 82, 83], "v9": 45, "valid": [4, 12, 14, 15, 16, 17, 21, 36, 44, 85], "valid_kei": 16, "validate_cach": [11, 21], "validate_kei": [11, 12, 13, 16, 17, 21], "valu": [0, 2, 4, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 24, 25, 26, 36, 43, 44, 45, 46, 48, 55, 57, 63, 71, 72, 73, 79, 80, 84], "valuabl": [0, 48, 63], "vanilla": 69, "var": [41, 42, 45, 62, 63, 79, 84, 85, 87], "vari": [9, 83], "variabl": [1, 2, 4, 11, 13, 14, 15, 17, 19, 20, 21, 28, 36, 39, 44, 47, 56, 58, 63, 69, 73, 80, 84, 85], "variant": [36, 63], "varieti": 45, "variou": [4, 10, 21, 31, 40, 43, 44, 47, 48, 50, 53, 55, 56, 58, 68, 73, 80, 84, 85, 87, 90], "variti": 68, "varnam": 14, "vast": 36, "vc": [36, 45], "ve": [31, 43, 48, 58, 59, 60, 63, 79, 80, 81, 82, 83], "veget": 73, "venv": [55, 69], "verbatim": [31, 43], "verbos": [84, 85, 87], "veri": [0, 4, 31, 36, 45, 46, 56, 82, 83], "verif": 85, "verifi": 85, "version": [0, 4, 14, 15, 17, 21, 22, 26, 31, 32, 33, 34, 35, 36, 38, 39, 43, 48, 50, 52, 55, 58, 59, 60, 62, 63, 65, 69, 72, 74, 79, 80, 81, 82, 83, 84, 85, 86, 87], "vertic": 43, "vi": 58, "via": [20, 39, 41, 45, 46, 48, 53, 55, 63, 85], "view": [9, 16, 32, 43, 55, 56], "virtu": 31, "virtual": [19, 21, 45, 80], "virtual_f": [11, 15], "virtualenv": 55, "visibl": [34, 43], "visit": [48, 50, 86], "visual": 10, "visualis": 52, "vital": 9, "vlock": 58, "void": 31, "volnam": 58, "volum": [86, 87], "w": 26, "wa": [0, 2, 10, 12, 13, 14, 15, 16, 17, 21, 22, 25, 26, 34, 40, 43, 44, 45, 48, 50, 52, 56, 58, 59, 63, 72, 73, 79, 81, 83, 84, 85, 87], "wai": [0, 1, 2, 4, 5, 7, 10, 14, 16, 20, 24, 31, 36, 40, 43, 44, 45, 48, 50, 52, 53, 55, 56, 57, 58, 59, 60, 62, 63, 64, 73, 79, 82, 83, 84, 85], "wait": [10, 31, 32, 50, 58, 62, 63, 72, 79, 80, 82, 83, 84, 85], "wake": 58, "walk": 26, "walkthrough": 92, "wall": [31, 59, 60, 80, 83], "wallclock": 85, "want": [0, 5, 17, 19, 31, 32, 34, 39, 41, 42, 43, 45, 47, 48, 50, 51, 52, 54, 55, 56, 58, 60, 69, 77, 80, 84, 85], "warn": [11, 14, 15, 25, 31, 32, 46, 50, 55, 56, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 85], "warning_token": 17, "warranti": [33, 34, 35, 38, 39, 45, 85], "watch": 48, "watermark": 85, "wc": 58, "we": [0, 2, 4, 5, 9, 10, 13, 16, 21, 26, 31, 32, 34, 35, 36, 41, 42, 43, 44, 45, 46, 47, 48, 50, 51, 53, 54, 55, 56, 58, 59, 60, 62, 63, 65, 69, 72, 73, 79, 80, 81, 82, 83, 84, 85], "weak": [2, 3], "webserv": 52, "websit": [50, 61, 69], "wednesdai": [31, 32, 58, 60, 62, 63, 79, 80, 81, 82, 83], "welcom": 69, "well": [0, 2, 3, 9, 21, 37, 43, 45, 48, 53, 55, 57, 64, 65, 69, 80, 88], "went": [48, 63], "were": [17, 20, 26, 33, 34, 40, 45, 46, 48, 55, 58, 83, 84], "wget": [42, 58], "what": [0, 4, 6, 10, 12, 13, 14, 15, 17, 20, 25, 26, 40, 43, 44, 47, 48, 50, 55, 56, 59, 60, 63, 66, 68, 79, 80, 82, 83, 84, 85], "whatev": [45, 47, 55, 84], "when": [0, 2, 3, 4, 5, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 31, 32, 34, 36, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 63, 64, 72, 73, 79, 80, 82, 83, 84, 85], "whenev": [0, 2, 10, 13, 17, 20, 21, 36, 43, 45, 50, 56, 57, 73, 82], "where": [4, 5, 12, 13, 16, 17, 19, 20, 21, 22, 24, 30, 32, 34, 36, 43, 44, 45, 48, 50, 54, 56, 59, 60, 63, 79, 80, 81, 82, 83, 84, 85], "wherea": [44, 48, 73], "wherev": 39, "whether": [0, 3, 13, 14, 16, 17, 19, 20, 21, 24, 26, 33, 34, 39, 43, 45, 48, 52, 59, 63, 72, 76, 84, 85], "which": [0, 1, 2, 3, 4, 5, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 25, 26, 31, 32, 33, 34, 36, 37, 40, 41, 42, 43, 44, 45, 46, 47, 48, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 63, 64, 65, 68, 69, 72, 73, 77, 79, 80, 81, 82, 83, 84, 85, 87, 91], "while": [0, 2, 4, 12, 14, 17, 19, 21, 26, 30, 36, 43, 44, 45, 48, 54, 55, 56, 63, 79, 80, 84, 85], "whitelist": [9, 25, 28], "whitespac": [48, 85], "who": [0, 29, 41, 50, 85], "whoami": 58, "whoever": 0, "whoi": 58, "whole": [40, 45, 48, 84], "whom": 0, "whose": [0, 15, 34, 40, 84], "why": [0, 5, 15, 48], "wide": [44, 45, 48, 65, 69, 79, 82, 85], "wider": 9, "widget": 53, "wiki": [41, 42], "wildcard": 84, "willing": 0, "win32": 0, "window": [36, 59, 62], "wip": 0, "wish": [29, 32, 53], "within": [0, 4, 9, 12, 14, 17, 19, 20, 21, 24, 25, 30, 43, 45, 48, 50, 53, 55, 56, 58, 83, 85, 87], "without": [2, 4, 9, 13, 14, 16, 17, 20, 21, 25, 31, 33, 34, 35, 38, 39, 40, 43, 45, 46, 48, 50, 55, 56, 59, 63, 68, 80, 84, 85, 87], "wizard": 0, "won": [0, 80], "word": [44, 48, 56], "wordi": 4, "work": [0, 1, 3, 8, 9, 11, 12, 13, 17, 19, 20, 22, 29, 31, 32, 36, 43, 45, 48, 50, 53, 55, 56, 57, 58, 60, 64, 68, 71, 79, 82, 83, 84, 86, 89], "work_dir": 20, "work_serializer_clears_time_cach": [31, 32, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83], "worker": [8, 85, 87], "workflow": [0, 45, 50, 65], "workload": 52, "workspac": [3, 12, 19, 21, 34, 45, 48, 53, 59, 65, 66, 68, 71, 85, 88, 91], "workspace_hello": [32, 64], "workspace_libhello": 31, "workspace_subproject": 64, "workspacedir": 85, "world": [32, 43, 58, 59, 62, 63, 81], "wors": 52, "worth": [0, 36, 48], "worthwhil": 50, "would": [0, 14, 16, 17, 26, 31, 34, 36, 41, 43, 45, 52, 54, 55, 58, 60, 62, 63, 78, 83], "wrap": [26, 48, 50], "wrapper": [1, 14, 17], "writabl": 9, "write": [0, 9, 12, 13, 19, 20, 26, 33, 34, 35, 38, 39, 41, 42, 45, 48, 55, 67, 69, 85], "written": [9, 26, 31, 32, 45, 48, 50, 52, 55, 56, 58, 59, 60, 62, 63, 64, 79, 80, 81, 82, 83, 84], "wrong": [48, 58], "wsl": 0, "www": [26, 33, 34, 35, 38, 39, 41, 42, 45, 85], "x": [1, 16, 17, 22, 54, 55, 62, 63, 79], "x86": [43, 45, 52, 69, 87], "x86_32": 65, "x86_64": [32, 54, 58, 62, 63, 65, 79, 80, 82, 83], "xarg": 58, "xattr": 9, "xdff": 50, "xdg_cache_hom": [69, 74, 85], "xdg_config_hom": 85, "xdg_runtime_dir": 45, "xmlwf": 58, "xsl": 41, "xxd": 58, "xz": [16, 32, 42, 58, 62, 63, 79, 80, 82, 83, 84], "xzcat": 58, "y": 54, "y7x5pnm": 79, "yaml": [4, 11, 12, 13, 14, 15, 17, 20, 21, 22, 28, 29, 30, 44, 45, 46, 48, 55, 56, 57, 63, 65, 67, 68, 73, 79, 80, 85], "ye": [32, 58, 60, 62, 63, 79, 80, 81, 82, 83], "yelp": 41, "yet": [4, 14, 19, 85, 87], "yield": [14, 21, 24, 26], "yml": [86, 87], "you": [0, 1, 5, 9, 12, 16, 17, 26, 30, 31, 32, 33, 34, 35, 36, 38, 39, 40, 43, 44, 45, 48, 49, 50, 52, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 68, 69, 74, 76, 77, 79, 80, 81, 83, 84, 85, 86, 89], "your": [0, 1, 9, 12, 13, 15, 16, 17, 20, 21, 28, 31, 35, 36, 40, 41, 42, 43, 44, 45, 46, 48, 50, 52, 55, 56, 57, 58, 60, 62, 63, 64, 65, 69, 70, 71, 72, 74, 76, 77, 79, 82, 83, 85, 92], "yourself": [50, 56], "z": 26, "zcompdump": 69, "zero": [9, 17, 26, 68, 84], "zfunc": 69, "zip": 73, "zlib": 46, "zoneinfo": 45, "zprezto": 69, "zsh_custom": 69, "zshrc": 69}, "titles": ["Contributing", "BuildStream and Docker", "Cache keys", "Caches", "Data model", "Dependency model", "Overview of modules", "Overview of program flow", "Remote execution", "Sandboxing", "Scheduler", "buildstream package", "BuildElement - Abstract class for build elements", "DownloadableFileSource - Abstract class for sources downloaded from a URI", "Element - Base element class", "Exceptions - API for Error Handling", "Node - Parsed YAML configuration", "Plugin - Base plugin class", "buildstream.sandbox package", "Sandbox - The build sandbox", "ScriptElement - Abstract class for scripting elements", "Source - Base source class", "SourceMirror - Base source mirror class", "buildstream.storage package", "Directory - Interfacing with files", "Foundation types", "Utilities", "Additional writings", "Project format", "Plugin API reference", "Plugin specific documentation", "2. Strict mode", "1. Workspaces", "compose - Compose the output of multiple elements", "filter - Extract a subset of files from another element", "import - Import sources directly", "junction - Integrate subprojects", "link - Link elements", "manual - Manual build element", "script - Run scripts to create output", "stack - Symbolic Element for dependency grouping", "Creating and using a git mirror", "Creating and using a tar mirror", "Declaring elements", "Introduction", "Project configuration", "The project.refs file", "Builtin public data", "Coding guidelines", "Generating protocol buffers", "Making releases", "Managing data files", "Measuring performance", "Contributing to the UI", "Updating BuildStream\u2019s Python dependencies", "Using the test suite", "Writing documentation", "Adding core plugins", "1. Composition", "2. Filtering", "3. Overlapping files", "BuildStream documentation", "1. Junction elements", "3. Subproject includes", "2. Workspaces and subprojects", "About", "Architecture", "Reference", "Glossary", "Installing from Source", "Porting guide", "Using", "Porting command line usage", "Porting the project format", "Porting the buildstream.conf", "local - stage local files and directories", "remote - stage files from remote urls", "tar - stage files from tar archives", "workspace - stage an opened workspace directory", "3. Using the autotools element", "5. Optionality and directives", "1. Your first project", "4. Integration commands", "2. Running commands", "Commands", "User configuration", "Configuring Cache Servers", "Remote Execution Servers", "Developing", "Examples", "Handling files", "Combining projects", "Getting started"], "titleterms": {"": 54, "1": [41, 42, 44, 74], "2": [41, 42, 44, 74], "3": [41, 42, 44], "4": [41, 42, 44], "5": [41, 42, 44], "6": [41, 42], "8": 48, "A": 62, "The": [12, 19, 46, 53, 73, 81], "abi": 54, "about": 65, "abstract": [12, 13, 14, 17, 20, 21, 22, 48], "access": [0, 9, 21], "accessor": 48, "activ": 50, "ad": [0, 54, 55, 56, 57], "add": [57, 81], "addit": 27, "address": [3, 43], "against": 59, "alia": 73, "alias": 45, "alpin": [58, 79, 82, 83], "alreadi": 31, "alwai": 48, "an": [56, 73, 78], "anoth": 34, "api": [15, 29, 48], "append": 44, "approxim": 48, "architectur": [45, 66], "archiv": 77, "argument": 48, "artifact": [3, 8, 45, 59, 60, 73, 84, 85], "assembl": 12, "assert": 44, "assign": 45, "attribut": [73, 84, 85], "auth": 13, "authent": 85, "autotool": 79, "avoid": 48, "base": [14, 17, 21, 22, 58, 79, 82, 83], "bash": 69, "basic": [43, 46], "behavior": 46, "benchmark": 52, "binari": [12, 54, 69], "bonu": 41, "boolean": 45, "both": 63, "branch": 0, "bst": [31, 58, 59, 63, 72, 73, 79, 80, 81, 82, 83, 84], "bst_profil": 52, "bsterror": 48, "buffer": 49, "bug": 0, "build": [2, 8, 12, 14, 19, 31, 32, 38, 43, 55, 56, 58, 62, 63, 69, 72, 79, 80, 81, 82, 83, 84, 85], "buildbarn": 86, "buildbox": 69, "buildel": 12, "buildstream": [1, 11, 18, 23, 43, 52, 54, 56, 61, 65, 69, 73, 74, 87], "built": [12, 21, 31, 59], "builtin": [44, 45, 47], "bump": 54, "bundl": 72, "c": [31, 59, 80, 83], "ca": 3, "cach": [2, 3, 8, 45, 73, 74, 85, 86], "cachekei": 57, "call": 48, "can": [9, 41, 42, 65, 73], "chang": [32, 50], "check": 40, "checkout": [69, 72, 84], "ci": [0, 54], "class": [12, 13, 14, 16, 17, 20, 21, 22, 48], "close": [32, 84], "code": [32, 48, 49, 55], "codebas": 48, "combin": 91, "command": [12, 45, 47, 53, 56, 72, 73, 82, 83, 84, 85], "commit": 0, "committ": 0, "common": 45, "commonli": 84, "compat": 54, "complet": 69, "compos": [33, 54], "composit": [4, 44, 58], "condit": [44, 63], "conf": [41, 59, 63, 73, 74, 79, 80, 81, 82, 83], "config": 43, "configur": [12, 16, 17, 41, 42, 44, 45, 63, 73, 74, 85, 86, 87], "configure_sandbox": 12, "consist": 48, "constraint": 45, "contain": [41, 42], "content": [3, 58, 81, 84], "context": 4, "contribut": [0, 53], "control": 85, "convent": 48, "core": [57, 73], "coverag": 55, "creat": [39, 41, 42, 73, 81], "cross": [43, 64], "custom": 45, "cython": 54, "data": [4, 13, 47, 51, 82], "declar": [43, 44, 58, 73, 81], "default": [44, 45, 85], "delet": 84, "depend": [5, 12, 40, 43, 54, 55, 69], "deploi": 40, "deprec": 45, "detail": 54, "develop": 88, "devic": 9, "direct": [44, 80], "directli": [35, 59], "directori": [12, 24, 41, 42, 44, 74, 75, 78, 85], "displai": 53, "distribut": 69, "do": [9, 65], "doc": 56, "docker": [1, 54], "document": [30, 48, 56, 57, 61], "doe": 65, "download": 13, "downloadablefilesourc": 13, "duplic": 45, "dynam": 31, "element": [4, 9, 12, 14, 20, 30, 31, 33, 34, 36, 37, 38, 40, 43, 45, 58, 59, 62, 63, 73, 79, 80, 81, 82, 83], "enhanc": 48, "enumer": 45, "environ": [43, 45, 69], "error": [15, 48], "essenti": 45, "everyth": 31, "exampl": [56, 58, 62, 73, 87, 89], "except": [15, 48], "execut": [8, 85, 87], "express": 43, "extend": 57, "extens": 8, "extern": [30, 73], "extra": 13, "extract": 34, "featur": 0, "fetch": [21, 41, 42, 72, 84, 85], "file": [0, 24, 31, 34, 45, 46, 48, 51, 59, 60, 75, 76, 77, 80, 83, 85, 90], "filenam": 74, "filesystem": [9, 87], "filter": [34, 59], "first": 81, "fix": [0, 52], "flag": 45, "flow": 7, "format": [28, 46, 55, 56, 73], "foundat": 25, "framework": 52, "from": [13, 34, 41, 42, 63, 69, 76, 77], "function": [12, 21], "funki": 63, "further": [0, 41, 42, 62], "gener": [1, 21, 49], "get": [65, 92], "git": [41, 69], "github": 54, "global": 85, "glossari": 68, "grant": 0, "greet": 80, "group": 40, "guid": [56, 69, 70], "guidelin": [48, 56], "handl": [15, 48, 90], "hello": [31, 59, 60, 63, 79, 80, 81, 82, 83], "host": 45, "how": [0, 65], "html": 56, "http": [13, 41], "i": [0, 31, 65], "imag": 1, "implement": [12, 86, 87], "import": [35, 48], "includ": [44, 63, 80], "increment": 32, "index": 57, "individu": 21, "inform": [0, 53], "ini": 54, "init": [72, 84], "input": 87, "insid": 1, "instal": [12, 55, 69, 74], "instanc": [36, 48], "integr": [36, 47, 82], "interact": 45, "interfac": [24, 53], "intermedi": 40, "intern": [10, 21, 45], "introduc": 58, "introduct": [9, 44], "issu": [0, 52], "job": 10, "junction": [36, 43, 45, 62, 63, 64, 73], "kei": [2, 21], "kind": 43, "known": 86, "level": 84, "libhello": [31, 59, 60, 82], "lighttpd": [41, 42], "line": [53, 72, 85], "link": [36, 37], "linter": 55, "linux": 9, "list": [44, 58, 84, 85], "load": [21, 45, 73], "local": [45, 73, 75], "locat": 12, "log": [84, 85], "look": [79, 82], "make": [32, 50], "makefil": [31, 59, 60, 80, 83], "man": 56, "manag": [21, 51], "manual": 38, "map": 45, "mask": 45, "measur": 52, "messag": 0, "metadata": 9, "method": [12, 14, 17, 21, 22, 48], "migrat": 73, "minim": 48, "minimum": 45, "mirror": [22, 41, 42, 45, 85], "miscellan": [14, 73], "mitig": 60, "mode": [31, 63], "model": [4, 5, 9], "modifi": 31, "modul": [6, 48], "multipl": [33, 36], "mung": 60, "mutat": 48, "name": [0, 43, 45, 48, 73, 84], "need": 48, "nest": 36, "network": 9, "new": [0, 21, 54], "node": 16, "non": 2, "normal": 63, "note": [9, 69], "o": 45, "object": 21, "observ": [31, 55, 59, 81], "onli": [58, 73], "open": [31, 32, 72, 78, 84], "option": [36, 45, 63, 80, 85], "order": 48, "organ": 0, "other": [8, 9, 36], "out": 40, "output": [33, 39, 56, 81], "outsid": 8, "overlap": [47, 60, 73], "overrid": [36, 43, 44, 45, 85], "overview": [6, 7, 31, 34, 36, 37, 58, 59, 60, 63, 79, 80, 82, 83], "overwrit": 44, "packag": [11, 18, 23, 54, 69], "page": 56, "parallel": 74, "paramet": [48, 84], "pars": 16, "part": 52, "patch": 0, "path": [24, 43, 45, 63], "pep": 48, "perform": 52, "permiss": 9, "phase": 14, "pip": [45, 73], "plan": [2, 85], "platform": [9, 87], "plugin": [17, 29, 30, 44, 45, 57, 73], "polici": 56, "port": [70, 72, 73, 74], "posix": 9, "post": 50, "pre": 50, "prepend": 44, "prerequisit": [41, 42], "previou": 21, "privat": 48, "process": 50, "profil": 52, "program": [7, 48, 58, 79, 80, 82, 83], "project": [4, 12, 28, 31, 36, 41, 42, 44, 45, 46, 58, 59, 60, 63, 73, 79, 80, 81, 82, 83, 85, 91], "properti": [45, 87], "protocol": 49, "public": [43, 47, 48, 82], "pull": [0, 72, 84], "push": [72, 84], "pypi": 73, "pyproject": 54, "python": 54, "queue": 10, "rais": 48, "read": [41, 42, 62], "recommend": 85, "redund": 43, "ref": [21, 45, 46], "refactor": 48, "refer": [14, 16, 17, 21, 22, 29, 43, 67], "regener": [49, 56], "releas": [50, 54], "remot": [8, 74, 76, 84, 85, 86, 87], "remov": 54, "request": 0, "requir": [50, 69, 87], "reset": [72, 84], "resourc": 10, "root": 87, "rule": [43, 45, 47, 58], "run": [1, 12, 31, 39, 55, 58, 59, 62, 63, 79, 80, 82, 83], "runtim": [43, 55, 58, 69], "sandbox": [8, 9, 18, 19, 43, 45], "save": 21, "schedul": [10, 85], "scope": 5, "script": [20, 39, 73], "scriptel": 20, "separ": 12, "server": [41, 42, 45, 85, 86, 87], "servic": 85, "session": 56, "set": [41, 42, 86], "shell": [45, 72, 84], "should": 65, "show": [72, 84], "simpl": 62, "some": [73, 81], "sourc": [3, 4, 12, 13, 21, 22, 30, 35, 36, 43, 45, 69, 72, 81, 84, 85], "sourcefetch": 21, "sourcemirror": [13, 22], "specif": [30, 52, 69, 85, 87], "specifi": 73, "split": [45, 47, 58], "src": [80, 83], "stack": [40, 48, 73], "stage": [12, 75, 76, 77, 78, 87], "start": [41, 42, 65, 92], "state": [21, 48], "static": 31, "storag": [3, 8, 23, 45], "strict": [2, 31, 85], "strip": [12, 73], "structur": [2, 4, 17, 31, 44, 48, 56, 58, 59, 60, 63, 79, 80, 82, 83], "style": 48, "subcommand": 84, "submit": 0, "submodul": [11, 18, 23], "subpackag": 11, "subproject": [36, 45, 63, 64], "subset": 34, "suit": 55, "summari": [31, 58, 59, 60, 63, 79, 80, 81, 82, 83], "support": 54, "suppress": 45, "surfac": 48, "symbol": [40, 48], "system": 9, "t": 9, "tail": 48, "tar": [41, 42, 77], "target": 45, "test": [41, 42, 55, 57], "time": 69, "toml": 54, "tool": 52, "top": 84, "toplevel": [40, 63], "tox": 54, "track": [21, 72, 84, 85], "transient": 48, "type": [2, 16, 25, 43], "ui": 53, "uniqu": 21, "up": [41, 42, 86], "updat": [54, 57], "uri": 13, "url": [21, 76], "us": [8, 31, 40, 41, 42, 43, 55, 58, 59, 60, 63, 65, 71, 73, 79, 80, 82, 83, 84], "usag": 72, "user": [9, 56, 85], "util": 26, "valu": 85, "variabl": [12, 43, 45, 48, 79], "version": [45, 54, 73], "vertic": 48, "virtual": 69, "want": 73, "warn": [17, 45], "what": [9, 65], "wheel": 54, "whitelist": [47, 60, 73], "why": 65, "window": 0, "work": [21, 41, 42, 65, 74, 85, 87], "workflow": 54, "workspac": [4, 31, 32, 64, 72, 78, 84], "world": [79, 80, 82, 83], "write": [27, 56], "yaml": 16, "yml": 54, "you": [41, 42, 73], "your": [32, 73, 81], "zsh": 69}}) \ No newline at end of file diff --git a/sources/local.html b/sources/local.html new file mode 100644 index 000000000..d8fc7e693 --- /dev/null +++ b/sources/local.html @@ -0,0 +1,144 @@ + + + + + + + + + local - stage local files and directories — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

local - stage local files and directories

+

Usage:

+
# Specify the local source kind
+kind: local
+
+# Specify the project relative path to a file or directory
+path: files/somefile.txt
+
+
+

See built-in functionality doumentation for +details on common configuration options for sources.

+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/sources/remote.html b/sources/remote.html new file mode 100644 index 000000000..fe9e56c37 --- /dev/null +++ b/sources/remote.html @@ -0,0 +1,157 @@ + + + + + + + + + remote - stage files from remote urls — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

remote - stage files from remote urls

+

Usage:

+
# Specify the remote source kind
+kind: remote
+
+# Optionally specify a relative staging filename.
+# If not specified, the basename of the url will be used.
+# filename: customfilename
+
+# Optionally specify whether the downloaded file should be
+# marked executable.
+# executable: true
+
+# Specify the url. Using an alias defined in your project
+# configuration is encouraged. 'bst source track' will update the
+# sha256sum in 'ref' to the downloaded file's sha256sum.
+url: upstream:foo
+
+# Specify the ref. It's a sha256sum of the file you download.
+ref: 6c9f6f68a131ec6381da82f2bff978083ed7f4f7991d931bfa767b7965ebc94b
+
+
+

See built-in functionality doumentation for +details on common configuration options for sources.

+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/sources/tar.html b/sources/tar.html new file mode 100644 index 000000000..d3532529e --- /dev/null +++ b/sources/tar.html @@ -0,0 +1,166 @@ + + + + + + + + + tar - stage files from tar archives — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

tar - stage files from tar archives

+

Host dependencies:

+
+
    +

  • lzip (for .tar.lz files)

    • +
+
+

Usage:

+
# Specify the tar source kind
+kind: tar
+
+# Specify the tar url. Using an alias defined in your project
+# configuration is encouraged. 'bst source track' will update the
+# sha256sum in 'ref' to the downloaded file's sha256sum.
+url: upstream:foo.tar
+
+# Specify the ref. It's a sha256sum of the file you download.
+ref: 6c9f6f68a131ec6381da82f2bff978083ed7f4f7991d931bfa767b7965ebc94b
+
+# Specify a glob pattern to indicate the base directory to extract
+# from the tarball. The first matching directory will be used.
+#
+# Note that this is '*' by default since most standard release
+# tarballs contain a self named subdirectory at the root which
+# contains the files one normally wants to extract to build.
+#
+# To extract the root of the tarball directly, this can be set
+# to an empty string.
+base-dir: '*'
+
+
+

See built-in functionality doumentation for +details on common configuration options for sources.

+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/sources/workspace.html b/sources/workspace.html new file mode 100644 index 000000000..dc9b1b773 --- /dev/null +++ b/sources/workspace.html @@ -0,0 +1,121 @@ + + + + + + + + + workspace - stage an opened workspace directory — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

workspace - stage an opened workspace directory

+

Usage:

+

The workspace plugin must not be directly used. This plugin is used as the +kind for a synthetic node representing the sources of an element with an open +workspace. The node constructed would be specified as follows:

+
# Specify the workspace source kind
+kind: workspace
+
+# Specify the absolute path to the directory
+path: /path/to/workspace
+
+
+
+ + +
+
+
+ +
+ +
+

© Copyright 2017-2022, The Apache Software Foundation.

+
+ + Built with Sphinx using a + theme + provided by Read the Docs. + + +
+
+
+
+
+ + + + \ No newline at end of file diff --git a/tutorial/autotools.html b/tutorial/autotools.html new file mode 100644 index 000000000..6042a8c1f --- /dev/null +++ b/tutorial/autotools.html @@ -0,0 +1,490 @@ + + + + + + + + + 3. Using the autotools element — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

3. Using the autotools element

+

In the last chapter we observed how the +manual element works, allowing one to specify and +run commands manually in the process of constructing an artifact.

+

In this chapter, we’ll go over a mostly automated build of a similar +hello world example. We will observe how our configurations of the +autotools element translate to configurations +on the manual element, and observe how +variable substitution works.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/autotools +subdirectory.

+
+
+

3.1. Overview

+

Instead of using the local source as we have been using +in the previous examples, we’re going to use a tar source +this time to obtain the automake release tarball directly from the upstream +hosting.

+

In this example we’re going to build the example program included in the +upstream automake tarball itself, and we’re going to use the automated +autotools build element to do so.

+
+
+

3.2. Project structure

+
+

3.2.1. project.conf

+
# Unique project name
+name: autotools
+
+# Minimum required BuildStream version
+min-version: 2.0
+
+# Subdirectory where elements are stored
+element-path: elements
+
+# Define some aliases for the tarballs we download
+aliases:
+  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
+  gnu: http://ftpmirror.gnu.org/gnu/automake/
+
+plugins:
+- origin: pip
+  package-name: buildstream-plugins
+  elements:
+  - autotools
+
+
+

Like the last project.conf, we’ve +added another source alias for gnu, the location +from which we’re going to download the automake tarball.

+
+
+

3.2.2. elements/base/alpine.bst and elements/base.bst

+

The alpine base and base stack element are defined in the +same way as in the last chapter: Running commands.

+
+
+

3.2.3. elements/hello.bst

+
kind: autotools
+description: |
+
+    Hello world example from automake
+
+variables:
+
+  # The hello world example lives in the doc/amhello folder.
+  #
+  # Set the %{command-subdir} variable to that location
+  # and just have the autotools element run its commands there.
+  #
+  command-subdir: doc/amhello
+
+sources:
+- kind: tar
+  url: gnu:automake-1.16.tar.gz
+  ref: 80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f
+
+depends:
+- base.bst
+
+
+

In this case, we haven’t touched the element’s config section +at all, instead we just slightly override the bahavior of the +autotools build element by overriding +the command-subdir variable

+
+

3.2.3.1. Looking at variables

+

Let’s take a moment and observe how element composition works with variables.

+

As the documentation mentions:

+
    +

  • The initial settings of the project.conf variables are setup +using BuildStream’s builtin defaults.

    • +

  • After this, your local project.conf may override some variables +on a project wide basis. Those will in turn be overridden by any +defaults provided by element classes, such as the variables set in the +documentation of the autotools build element. +The variables you set in your final <element.bst> element declarations, +will have the final say on the value of a particular variable.

    • +

  • Finally, the variables, which may be composed of other variables, +are resolved after all composition has taken place.

    • +
+

The variable we needed to override was command-subdir, which is an +automatic variable provided by the BuildElement +abstract class. This variable simply instructs the BuildElement +in which subdirectory of the %{build-root} to run its commands in.

+

One can always display the resolved set of variables for a given +element’s configuration using bst show:

+ +
+user@host:~/autotools$ bst show --deps none --format "%{vars}" hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888140.066774     297 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+prefix: /usr
+exec_prefix: /usr
+bindir: /usr/bin
+sbindir: /usr/sbin
+libexecdir: /usr/libexec
+datadir: /usr/share
+sysconfdir: /etc
+sharedstatedir: /usr/com
+localstatedir: /var
+lib: lib
+libdir: /usr/lib
+debugdir: /usr/lib/debug
+includedir: /usr/include
+docdir: /usr/share/doc
+infodir: /usr/share/info
+mandir: /usr/share/man
+build-root: /buildstream/autotools/hello.bst
+conf-root: .
+install-root: /buildstream-install
+strip-binaries:
+project-name: autotools
+max-jobs: 4
+autogen: "export NOCONFIGURE=1;\n\nif [ -x ./configure ]; then true;\nelif [ -x ./autogen
+  ]; then ./autogen;\nelif [ -x ./autogen.sh ]; then ./autogen.sh;\nelif [ -x ./bootstrap
+  ]; then ./bootstrap;\nelif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;\nelse autoreconf
+  -ivf .;\nfi"
+conf-global:
+conf-local:
+conf-cmd: ./configure
+conf-args: "--prefix=/usr \\\n--exec-prefix=/usr \\\n--bindir=/usr/bin \\\n--sbindir=/usr/sbin
+  \\\n--sysconfdir=/etc \\\n--datadir=/usr/share \\\n--includedir=/usr/include \\\n
+  --libdir=/usr/lib \\\n--libexecdir=/usr/libexec \\\n--localstatedir=/var \\\n--sharedstatedir=/usr/com
+  \\\n--mandir=/usr/share/man \\\n--infodir=/usr/share/info  "
+configure: "./configure --prefix=/usr \\\n--exec-prefix=/usr \\\n--bindir=/usr/bin
+  \\\n--sbindir=/usr/sbin \\\n--sysconfdir=/etc \\\n--datadir=/usr/share \\\n--includedir=/usr/include
+  \\\n--libdir=/usr/lib \\\n--libexecdir=/usr/libexec \\\n--localstatedir=/var \\\n
+  --sharedstatedir=/usr/com \\\n--mandir=/usr/share/man \\\n--infodir=/usr/share/info\
+  \  "
+make-args:
+make-install-args: ' DESTDIR="/buildstream-install" install'
+make: 'make '
+make-install: make -j1  DESTDIR="/buildstream-install" install
+remove-libtool-modules: false
+remove-libtool-libraries: false
+delete-libtool-archives: "if false || false; then\n  find \"/buildstream-install\"\
+  \ -name \"*.la\" -print0 | while read -d '' -r file; do\n    if grep '^shouldnotlink=yes$'
+  \"${file}\" &>/dev/null; then\n      if false; then\n        echo \"Removing ${file}.\"\
+  \n        rm \"${file}\"\n      else\n        echo \"Not removing ${file}.\"\n \
+  \     fi\n    else\n      if false; then\n        echo \"Removing ${file}.\"\n \
+  \       rm \"${file}\"\n      else\n        echo \"Not removing ${file}.\"\n   \
+  \   fi\n    fi\n  done\nfi"
+command-subdir: doc/amhello
+element-name: hello.bst
+
+
+

As an exercise, we suggest that you modify the hello.bst +element to set the prefix like so:

+
variables:
+  prefix: "/opt"
+
+
+

And rerun the above bst show command to observe how this +changes the output.

+

Observe where the variables are declared in the builtin defaults and autotools element +documentation, and how overriding these effects the resolved set of variables.

+
+
+
+
+

3.3. Using the project

+
+

3.3.1. Build the hello.bst element

+

To build the project, run bst build in the +following way:

+ +
+user@host:~/autotools$ bst build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888140.476894     325 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:15:40
+    Project:       autotools (/home/user/buildstream/doc/examples/autotools)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-y7x5pnms/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-y7x5pnms
+    Log Files:               /home/user/buildstream/doc/run-bst-y7x5pnms/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-y7x5pnms/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-y7x5pnms/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: autotools
+
+    Element Plugins
+        autotools: python package 'buildstream-plugins 2.2.0' at: /home/user/buildstream/.tox/docs/lib/python3.12/site-packages
+        stack:     core plugin
+        import:    core plugin
+
+    Source Plugins
+        tar: core plugin
+
+Pipeline
+fetch needed 6a78c03097648e558c803c20d8d20f61d3e87e3e616f3a45923704cfe95019af base/alpine.bst 
+     waiting 644e964abe673c8a8723cf5a3fceeeda44a7bea364f55604fc8132af44a4fb59 base.bst 
+     waiting ffeab61203b5940714c33b1cc29822b9391b9fea5876faef23c024bf6acd99cf hello.bst 
+===============================================================================
+[--:--:--][6a78c030][   fetch:base/alpine.bst               ] START   autotools/base-alpine/6a78c030-fetch.20241106-101540.log
+[--:--:--][6a78c030][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[--:--:--][644e964a][   fetch:base.bst                      ] START   autotools/base/644e964a-fetch.20241106-101540.log
+[00:00:00][644e964a][   fetch:base.bst                      ] SUCCESS autotools/base/644e964a-fetch.20241106-101540.log
+[00:00:01][6a78c030][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:08][6a78c030][   fetch:base/alpine.bst               ] SUCCESS autotools/base-alpine/6a78c030-fetch.20241106-101540.log
+[--:--:--][6a78c030][   build:base/alpine.bst               ] START   autotools/base-alpine/6a78c030-build.20241106-101548.log
+[--:--:--][6a78c030][   build:base/alpine.bst               ] START   Staging sources
+[00:00:00][6a78c030][   build:base/alpine.bst               ] SUCCESS Staging sources
+[--:--:--][6a78c030][   build:base/alpine.bst               ] START   Caching artifact
+[00:00:00][6a78c030][   build:base/alpine.bst               ] SUCCESS Caching artifact
+[00:00:00][6a78c030][   build:base/alpine.bst               ] SUCCESS autotools/base-alpine/6a78c030-build.20241106-101548.log
+[--:--:--][644e964a][   build:base.bst                      ] START   autotools/base/644e964a-build.20241106-101548.log
+[--:--:--][644e964a][   build:base.bst                      ] START   Caching artifact
+[00:00:00][644e964a][   build:base.bst                      ] SUCCESS Caching artifact
+[00:00:00][644e964a][   build:base.bst                      ] SUCCESS autotools/base/644e964a-build.20241106-101548.log
+[--:--:--][ffeab612][   build:hello.bst                     ] START   autotools/hello/ffeab612-build.20241106-101548.log
+[--:--:--][ffeab612][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][ffeab612][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][ffeab612][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][ffeab612][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][ffeab612][   build:hello.bst                     ] START   Staging sources
+[00:00:00][ffeab612][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][ffeab612][   build:hello.bst                     ] START   Running commands
+
+    export NOCONFIGURE=1;
+    
+    if [ -x ./configure ]; then true;
+    elif [ -x ./autogen ]; then ./autogen;
+    elif [ -x ./autogen.sh ]; then ./autogen.sh;
+    elif [ -x ./bootstrap ]; then ./bootstrap;
+    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
+    else autoreconf -ivf .;
+    fi
+    ./configure --prefix=/usr \
+    --exec-prefix=/usr \
+    --bindir=/usr/bin \
+    --sbindir=/usr/sbin \
+    --sysconfdir=/etc \
+    --datadir=/usr/share \
+    --includedir=/usr/include \
+    --libdir=/usr/lib \
+    --libexecdir=/usr/libexec \
+    --localstatedir=/var \
+    --sharedstatedir=/usr/com \
+    Message contains 23 additional lines
+
+[00:00:03][ffeab612][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][ffeab612][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][ffeab612][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:03][ffeab612][   build:hello.bst                     ] SUCCESS autotools/hello/ffeab612-build.20241106-101548.log
+[00:00:12][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 2, skipped 1, failed 0 
+    Build Queue: processed 3, skipped 0, failed 0
+
+
+
+

3.3.2. Run the hello world program

+

We probably know by now what’s going to happen, but let’s run +the program we’ve compiled anyway using bst shell:

+ +
+user@host:~/autotools$ bst shell hello.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888153.142350    1408 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][ffeab612][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][ffeab612][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][ffeab612][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][ffeab612][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][ffeab612][    main:hello.bst                     ] STATUS  Running command
+
+    hello
+
+Hello World!
+This is amhello 1.0.
+
+
+
+
+

3.4. Summary

+

Now we’ve used an external build element, +from the buildstream-plugins package and we’ve taken a look into +how variables work.

+

When browsing the build elements in their +respective documentation, we are now equipped with a good idea of what an element +is going to do, based on their default YAML configuration and any configurations +we have in our project. We can also now observe what variables are in effect +for the build of a given element, using bst show.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/tutorial/directives.html b/tutorial/directives.html new file mode 100644 index 000000000..8f1bc0b2a --- /dev/null +++ b/tutorial/directives.html @@ -0,0 +1,674 @@ + + + + + + + + + 5. Optionality and directives — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

5. Optionality and directives

+

In this chapter we’re going to go over some of the more flexible constructs +which BuildStream offers for optionality, and +show how we can use directives in the BuildStream +YAML format.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/directives +subdirectory.

+
+
+

5.1. Overview

+

This chapter’s example will build another hello.c program which much +resembles the program in the running commands example, +but here we’re going to make the greeting string configurable using the C +preprocessor.

+

We’ll be compiling the following C file:

+
+

5.1.1. files/src/hello.c

+
/*
+ * hello.c - Simple hello world program
+ */
+#include <stdio.h>
+
+int main(int argc, char *argv[])
+{
+  printf(GREETING_MESSAGE);
+  return 0;
+}
+
+
+

And we’re going to build it using make, using the following Makefile:

+
+
+

5.1.2. files/src/Makefile

+
# Sample makefile for hello.c
+#
+.PHONY: all install
+
+all: hello
+
+install:
+	install -d ${DESTDIR}${PREFIX}/bin
+	install -m 755 hello ${DESTDIR}${PREFIX}/bin
+
+hello: hello.c
+	$(CC) -DGREETING_MESSAGE="\"${GREETING}\"" -Wall -o $@ $<
+
+
+

Notice the addition of -DGREETING_MESSAGE="\"${GREETING}\"" in the above +Makefile, this will allow us to configure the greeting message from the +hello.bst element declaration.

+

We will need to add support to our project for optionality, and we’ll +have to make conditional statements to resolve what kind of greeting +we want from the hello world program.

+
+
+
+

5.2. Project structure

+

Since this project has much the same structure as the +running commands chapter did, we won’t go over all of +these elements in detail. Instead let’s focus on the addition of the new +project options in project.conf, the added +file in the include/ project subdirectory, and how these come together +in the the hello.bst element.

+
+

5.2.1. project.conf

+
# Unique project name
+name: directives
+
+# Minimum required BuildStream version
+min-version: 2.0
+
+# Subdirectory where elements are stored
+element-path: elements
+
+# Define an alias for our alpine tarball
+aliases:
+  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
+
+# Define an option for this project
+#
+options:
+  flavor:
+    type: enum
+    description: Flavor of the greeting in the hello world program
+    values:
+    - normal
+    - somber
+    - excited
+    default: normal
+
+
+

Here, our project.conf declares a project option called flavor, and this +will inform what kind of greeting message we want to use when building the project.

+
+
+

5.2.2. elements/hello.bst

+
kind: manual
+description: |
+
+  A hello world program with a custom greeting message
+
+# Depend on the base system
+depends:
+- base.bst
+
+# Stage the files/src directory for building
+sources:
+  - kind: local
+    path: files/src
+
+# This include file defines the %{greeting} variable used below
+variables:
+  (@): include/greeting.bst
+
+# Now configure the commands to run
+config:
+
+  # This time we inform the Makefile of which greeting we want
+  build-commands:
+  - make PREFIX="%{prefix}" GREETING="%{greeting}"
+
+  install-commands:
+  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
+
+
+

Notice the (@) symbol we’ve added in the variables: section, this +symbol is used to invoke the include directive, +which can be useful for code sharing between elements or simply to improve readability.

+

In this case, we are compositing the content of include/greeting.bst into the +variables section of the element declaration, directives +can however be used virtually anywhere in the BuildStream YAML format.

+
+
+

5.2.3. include/greeting.bst

+
# We define the greeting message here conditionally
+(?):
+- flavor == "normal":
+    greeting: |
+
+      Hello world !
+
+- flavor == "somber":
+    greeting: |
+
+      Hey world.
+
+- flavor == "excited":
+    greeting: |
+
+      Howdy there, and what a world it is !
+
+
+

Here we can see the dictionary which will be composited into the variables: +section of the hello.bst element described above.

+

Note the usage of the (?) symbol at the toplevel of the YAML dictionary, +this is how we perform conditional statements +in the BuildStream YAML format.

+

This include file uses the flavor project option we declared in project.conf to +decide what value will end up being assigned to the %{greeting} variable, which +will ultimately be used in the hello.bst element.

+
+
+
+

5.3. Using the project

+

Now that we have a project which uses options and conditional statements, +lets build the project with a few different options and observe the outputs.

+
+

5.3.1. Building hello.bst element with options

+

Since the flavor option we’ve declared above +has a default, we can build it the first time using bst build +without any special command line options:

+ +
+user@host:~/directives$ bst build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888178.362886    2096 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:16:18
+    Project:       directives (/home/user/buildstream/doc/examples/directives)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-ejrdsw3w/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-ejrdsw3w
+    Log Files:               /home/user/buildstream/doc/run-bst-ejrdsw3w/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-ejrdsw3w/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-ejrdsw3w/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: directives
+
+    Project Options
+        flavor: normal
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Pipeline
+fetch needed 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst 
+     waiting 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst 
+     waiting 49a52215c02358a4bbbb632a74986fa23dc31a36ee8055feffbddc9e3c735d1a hello.bst 
+===============================================================================
+[--:--:--][4275015f][   fetch:base/alpine.bst               ] START   directives/base-alpine/4275015f-fetch.20241106-101618.log
+[--:--:--][4275015f][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[--:--:--][22843536][   fetch:base.bst                      ] START   directives/base/22843536-fetch.20241106-101618.log
+[00:00:00][22843536][   fetch:base.bst                      ] SUCCESS directives/base/22843536-fetch.20241106-101618.log
+[00:00:00][4275015f][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:06][4275015f][   fetch:base/alpine.bst               ] SUCCESS directives/base-alpine/4275015f-fetch.20241106-101618.log
+[--:--:--][4275015f][   build:base/alpine.bst               ] START   directives/base-alpine/4275015f-build.20241106-101625.log
+[--:--:--][4275015f][   build:base/alpine.bst               ] START   Staging sources
+[00:00:00][4275015f][   build:base/alpine.bst               ] SUCCESS Staging sources
+[--:--:--][4275015f][   build:base/alpine.bst               ] START   Caching artifact
+[00:00:00][4275015f][   build:base/alpine.bst               ] SUCCESS Caching artifact
+[00:00:00][4275015f][   build:base/alpine.bst               ] SUCCESS directives/base-alpine/4275015f-build.20241106-101625.log
+[--:--:--][22843536][   build:base.bst                      ] START   directives/base/22843536-build.20241106-101625.log
+[--:--:--][22843536][   build:base.bst                      ] START   Caching artifact
+[00:00:00][22843536][   build:base.bst                      ] SUCCESS Caching artifact
+[00:00:00][22843536][   build:base.bst                      ] SUCCESS directives/base/22843536-build.20241106-101625.log
+[--:--:--][49a52215][   build:hello.bst                     ] START   directives/hello/49a52215-build.20241106-101625.log
+[--:--:--][49a52215][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][49a52215][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][49a52215][   build:hello.bst                     ] START   Staging sources
+[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][49a52215][   build:hello.bst                     ] START   Running commands
+
+    make PREFIX="/usr" GREETING="Hello world !"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][49a52215][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS directives/hello/49a52215-build.20241106-101625.log
+[00:00:07][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 2, skipped 1, failed 0 
+    Build Queue: processed 3, skipped 0, failed 0
+
+

If we want to build the somber flavor, we just need to specify the +additional --option command line option to bst +in order to inform BuildStream of the options we want.

+ +
+user@host:~/directives$ bst --option flavor somber build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888185.826886    2157 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:16:25
+    Project:       directives (/home/user/buildstream/doc/examples/directives)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-ejrdsw3w/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-ejrdsw3w
+    Log Files:               /home/user/buildstream/doc/run-bst-ejrdsw3w/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-ejrdsw3w/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-ejrdsw3w/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: directives
+
+    Project Options
+        flavor: somber
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Pipeline
+      cached 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst 
+      cached 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst 
+   buildable 14183452db56642e12bc721cd3501237d397ac87b1f24988665d69177fe488cb hello.bst 
+===============================================================================
+[--:--:--][14183452][   build:hello.bst                     ] START   directives/hello/14183452-build.20241106-101625.log
+[--:--:--][14183452][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][14183452][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][14183452][   build:hello.bst                     ] START   Staging sources
+[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][14183452][   build:hello.bst                     ] START   Running commands
+
+    make PREFIX="/usr" GREETING="Hey world."
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][14183452][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:00][14183452][   build:hello.bst                     ] SUCCESS directives/hello/14183452-build.20241106-101625.log
+[00:00:00][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 0, skipped 3, failed 0 
+    Build Queue: processed 1, skipped 2, failed 0
+
+

Note that the --option option can be specified many times on the +bst command line, so as to support projects which have multiple +options.

+

Finally lets get the excited flavor built as well:

+ +
+user@host:~/directives$ bst --option flavor excited build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888186.448670    2213 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:16:26
+    Project:       directives (/home/user/buildstream/doc/examples/directives)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-ejrdsw3w/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-ejrdsw3w
+    Log Files:               /home/user/buildstream/doc/run-bst-ejrdsw3w/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-ejrdsw3w/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-ejrdsw3w/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: directives
+
+    Project Options
+        flavor: excited
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Pipeline
+      cached 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst 
+      cached 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst 
+   buildable 72169f19db218d37d9f9b2a8603c0b75dcd391d3a07e5ef900f95e6fdaaca1ba hello.bst 
+===============================================================================
+[--:--:--][72169f19][   build:hello.bst                     ] START   directives/hello/72169f19-build.20241106-101626.log
+[--:--:--][72169f19][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][72169f19][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][72169f19][   build:hello.bst                     ] START   Staging sources
+[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][72169f19][   build:hello.bst                     ] START   Running commands
+
+    make PREFIX="/usr" GREETING="Howdy there, and what a world it is !"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][72169f19][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS directives/hello/72169f19-build.20241106-101626.log
+[00:00:00][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 0, skipped 3, failed 0 
+    Build Queue: processed 1, skipped 2, failed 0
+
+

If you observe the cache keys above, you will notice that while +we have only three elements in the pipeline, counting base/alpine.bst, +base.bst and hello.bst, we have actually built five artifacts, +because the hello.bst is built differently each time, it has a +different cache key and is stored separately in the artifact cache.

+
+
+

5.3.2. Run the hello world program with options

+

Since the --option command line option to bst +is a main option, it can be used in any command.

+

Let’s run the hello program using bst shell +three times in a row, each time using a different option so we can +observe the results.

+ +
+user@host:~/directives$ bst shell hello.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888187.076727    2269 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][49a52215][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][49a52215][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][49a52215][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][49a52215][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][49a52215][    main:hello.bst                     ] STATUS  Running command
+
+    hello
+
+Hello world !
+
+ +
+user@host:~/directives$ bst --option flavor somber shell hello.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888187.604786    2313 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][14183452][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][14183452][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][14183452][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][14183452][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][14183452][    main:hello.bst                     ] STATUS  Running command
+
+    hello
+
+Hey world.
+
+ +
+user@host:~/directives$ bst --option flavor excited shell hello.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888188.130867    2357 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][72169f19][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][72169f19][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][72169f19][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][72169f19][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][72169f19][    main:hello.bst                     ] STATUS  Running command
+
+    hello
+
+Howdy there, and what a world it is !
+
+
+
+
+

5.4. Summary

+

In this chapter we’ve demonstrated how to declare project options, +how to use conditional directives, and also +how to use include directives.

+

To get more familliar with these concepts, you may want to explore the remaining +directives in the BuildStream YAML format, and also take +a look at the various types of project options that +are also supported.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/tutorial/first-project.html b/tutorial/first-project.html new file mode 100644 index 000000000..3564ae561 --- /dev/null +++ b/tutorial/first-project.html @@ -0,0 +1,371 @@ + + + + + + + + + 1. Your first project — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

1. Your first project

+

To get a feel for the basics, we’ll start with the most basic BuildStream project we +could think of.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/first-project +subdirectory.

+
+
+

1.1. Creating the project

+

First, lets create the project itself using the convenience bst init +command to create a little project structure:

+ +
+user@host:~/first-project$ bst init --project-name first-project
+
+Created project.conf at: /home/user/buildstream/doc/examples/first-project/project.conf
+
+

This will give you a project.conf which will look like this:

+
+

1.1.1. project.conf

+
# Unique project name
+name: first-project
+
+# Required BuildStream version
+min-version: 2.3
+
+# Subdirectory where elements are stored
+element-path: elements
+
+
+

The project.conf is a central point of configuration +for your BuildStream project.

+
+
+
+

1.2. Add some content

+

BuildStream processes directory trees as input and output, +so let’s just create a hello.world file for the project +to have.

+ +
+user@host:~/first-project$ touch hello.world
+
+
+
+

1.3. Declare the element

+

Here we’re going to declare a simple import element +which will import the hello.world file we’ve created in the previous step.

+

Create elements/hello.bst with the following content:

+
+

1.3.1. elements/hello.bst

+
kind: import
+
+# Use a local source to stage our file
+sources:
+- kind: local
+  path: hello.world
+
+# Configure the import element
+config:
+
+  # Place the content staged by sources at the
+  # root of the output artifact
+  target: /
+
+
+
+
+

1.3.2. The source

+

The local source used by the hello.bst element, +can be used to access files or directories which are stored in the same repository +as your BuildStream project. The hello.bst element uses the local +source to stage our local hello.world file.

+
+
+

1.3.3. The element

+

The import element can be used to simply add content +directly to the output artifacts. In this case, it simply takes the hello.world file +provided by its source and stages it directly to the artifact output root.

+
+

Tip

+

In this example so far we’ve used two plugins, the local +source and the import element.

+

You can always browse the documentation for all plugins in +the plugins section of the manual.

+
+
+
+
+

1.4. Build the element

+

In order to carry out the activities of the import element +we’ve declared, we’re going to have to ask BuildStream to build.

+

This process will collect all of the sources required for the specified hello.bst +and get the backing import element to generate an artifact +for us.

+ +
+user@host:~/first-project$ bst build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888203.763861    2831 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:16:43
+    Project:       first-project (/home/user/buildstream/doc/examples/first-project)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-hxcfj_9g/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-hxcfj_9g
+    Log Files:               /home/user/buildstream/doc/run-bst-hxcfj_9g/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-hxcfj_9g/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-hxcfj_9g/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: first-project
+
+    Element Plugins
+        import: core plugin
+
+    Source Plugins
+        local: core plugin
+
+Pipeline
+   buildable 8b3d535225335b4842b5d6b98ee6fdffc7340e190f33b79a2043f5c31e43fbde hello.bst 
+===============================================================================
+[--:--:--][8b3d5352][   fetch:hello.bst                     ] START   first-project/hello/8b3d5352-fetch.20241106-101643.log
+[00:00:00][8b3d5352][   fetch:hello.bst                     ] SUCCESS first-project/hello/8b3d5352-fetch.20241106-101643.log
+[--:--:--][8b3d5352][   build:hello.bst                     ] START   first-project/hello/8b3d5352-build.20241106-101643.log
+[--:--:--][8b3d5352][   build:hello.bst                     ] START   Staging sources
+[00:00:00][8b3d5352][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][8b3d5352][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][8b3d5352][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:00][8b3d5352][   build:hello.bst                     ] SUCCESS first-project/hello/8b3d5352-build.20241106-101643.log
+[00:00:00][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       1
+    Session:     1
+    Fetch Queue: processed 1, skipped 0, failed 0 
+    Build Queue: processed 1, skipped 0, failed 0
+
+

Now the artifact is ready.

+

Using bst show, we can observe that the artifact’s state, which was reported +as buildable in the bst build command above, has now changed to cached:

+ +
+user@host:~/first-project$ bst show hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888204.179921    2861 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+      cached 8b3d535225335b4842b5d6b98ee6fdffc7340e190f33b79a2043f5c31e43fbde hello.bst 
+
+
+
+

1.5. Observe the output

+

Now that we’ve finished building, we can checkout the output of the +artifact we’ve created using bst artifact checkout

+ +
+user@host:~/first-project$ bst artifact checkout --directory here hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888204.601576    2890 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][8b3d5352][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][8b3d5352][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][8b3d5352][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][8b3d5352][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][8b3d5352][    main:hello.bst                     ] START   Checking out files in 'here'
+[00:00:00][8b3d5352][    main:hello.bst                     ] SUCCESS Checking out files in 'here'
+
+

And observe that the file we expect is there:

+ +
+user@host:~/first-project$ ls ./here
+
+hello.world
+
+
+
+

1.6. Summary

+

In this section we’ve created our first BuildStream project from +scratch, but it doesnt do much.

+

We’ve observed the general structure of a BuildStream project, +and we’ve run our first build.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/tutorial/integration-commands.html b/tutorial/integration-commands.html new file mode 100644 index 000000000..7759a4ad4 --- /dev/null +++ b/tutorial/integration-commands.html @@ -0,0 +1,396 @@ + + + + + + + + + 4. Integration commands — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

4. Integration commands

+

Sometimes a software requires more configuration or processing than what is +performed at installation time, otherwise it will not run properly.

+

This is especially true in cases where a daemon or library interoperates +with third party extensions and needs to maintain a system wide cache whenever +its extensions are installed or removed; system wide font caches are an example +of this.

+

In these cases we use integration commands to +ensure that a runtime is ready to run after all of its components have been staged.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/integration-commands +subdirectory.

+
+
+

4.1. Overview

+

In this chapter, we’ll be exploring integration commands, +which will be our first look at public data.

+
+
+

4.2. Project structure

+
+

4.2.1. project.conf and elements/base.bst

+

The project.conf and base stack stack element are configured in the +same way as in the previous chapter: Running commands.

+
+
+

4.2.2. elements/base/alpine.bst

+
kind: import
+description: |
+
+    Alpine Linux base runtime
+
+sources:
+- kind: tar
+  url: alpine:integration-tests-base.v1.x86_64.tar.xz
+  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639
+
+#
+# Run ldconfig in the libdir before running anything
+#
+public:
+  bst:
+    integration-commands:
+    - ldconfig "%{libdir}"
+
+
+

This is the same base/alpine.bst we’ve seen in previous chapters, +except that we’ve added an integration command.

+

This informs BuildStream that whenever the output of this element is +expected to run, this command should be run first. In this case we +are simply running ldconfig as a precautionary measure, to ensure +that the runtime linker is ready to find any shared libraries we may +have added to %{libdir}.

+
+

4.2.2.1. Looking at public data

+

The integration commands used here is the first time +we’ve used any builtin public data.

+

Public data is a free form portion of an element’s configuration and +is not necessarily understood by the element on which it is declared, public +data is intended to be read by its reverse dependency elements.

+

This allows annotations on some elements to inform elements later in +the dependency chain about details of its artifact, or to suggest how +it should be processed.

+
+
+
+

4.2.3. elements/libhello.bst and elements/hello.bst

+

These are basically manual elements very similar to the ones we’ve +seen in the previous chapter: Running commands.

+

These produce a library and a hello program which uses the library, +we will consider these irrelevant to the topic and leave examination +of their sources +as an exercise for the reader.

+
+
+
+

4.3. Using the project

+
+

4.3.1. Build the hello.bst element

+

To build the project, run bst build in the +following way:

+ +
+user@host:~/integration-commands$ bst build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888205.705887    2950 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:16:45
+    Project:       integration-commands (/home/user/buildstream/doc/examples/integration-commands)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-tpwfzpoh/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-tpwfzpoh
+    Log Files:               /home/user/buildstream/doc/run-bst-tpwfzpoh/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-tpwfzpoh/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-tpwfzpoh/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: integration-commands
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Pipeline
+fetch needed 5ff937aaa74f04d961f639c06d636e4a93abbc3a8d289a10ae8084f1d800a2de base/alpine.bst 
+     waiting 0848bf1769eb7f774cfe146212b0cfe56f9d7b7320832c93a7823e50ab993138 base.bst 
+     waiting 3cf9edd00c109f063e4a0c30d826420e6d240e59ee10241d59b22a7883ecef4f libhello.bst 
+     waiting 76f82637c19f9595a18d2e480da027552cb53811e39b7481b2feb767f0d2311d hello.bst 
+===============================================================================
+[--:--:--][0848bf17][   fetch:base.bst                      ] START   integration-commands/base/0848bf17-fetch.20241106-101645.log
+[--:--:--][3cf9edd0][   fetch:libhello.bst                  ] START   integration-commands/libhello/3cf9edd0-fetch.20241106-101645.log
+[--:--:--][5ff937aa][   fetch:base/alpine.bst               ] START   integration-commands/base-alpine/5ff937aa-fetch.20241106-101645.log
+[--:--:--][5ff937aa][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:00][0848bf17][   fetch:base.bst                      ] SUCCESS integration-commands/base/0848bf17-fetch.20241106-101645.log
+[00:00:00][3cf9edd0][   fetch:libhello.bst                  ] SUCCESS integration-commands/libhello/3cf9edd0-fetch.20241106-101645.log
+[00:00:00][5ff937aa][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:06][5ff937aa][   fetch:base/alpine.bst               ] SUCCESS integration-commands/base-alpine/5ff937aa-fetch.20241106-101645.log
+[--:--:--][5ff937aa][   build:base/alpine.bst               ] START   integration-commands/base-alpine/5ff937aa-build.20241106-101652.log
+[--:--:--][5ff937aa][   build:base/alpine.bst               ] START   Staging sources
+[00:00:00][5ff937aa][   build:base/alpine.bst               ] SUCCESS Staging sources
+[--:--:--][5ff937aa][   build:base/alpine.bst               ] START   Caching artifact
+[00:00:00][5ff937aa][   build:base/alpine.bst               ] SUCCESS Caching artifact
+[00:00:00][5ff937aa][   build:base/alpine.bst               ] SUCCESS integration-commands/base-alpine/5ff937aa-build.20241106-101652.log
+[--:--:--][0848bf17][   build:base.bst                      ] START   integration-commands/base/0848bf17-build.20241106-101652.log
+[--:--:--][0848bf17][   build:base.bst                      ] START   Caching artifact
+[00:00:00][0848bf17][   build:base.bst                      ] SUCCESS Caching artifact
+[00:00:00][0848bf17][   build:base.bst                      ] SUCCESS integration-commands/base/0848bf17-build.20241106-101652.log
+[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   integration-commands/libhello/3cf9edd0-build.20241106-101652.log
+[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Staging dependencies at: /
+[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Staging dependencies at: /
+[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Integrating sandbox
+[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Integrating sandbox
+[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Staging sources
+[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Staging sources
+[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Running commands
+
+    make PREFIX="/usr"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Running commands
+[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Caching artifact
+[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Caching artifact
+[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS integration-commands/libhello/3cf9edd0-build.20241106-101652.log
+[--:--:--][76f82637][   build:hello.bst                     ] START   integration-commands/hello/76f82637-build.20241106-101653.log
+[--:--:--][76f82637][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][76f82637][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][76f82637][   build:hello.bst                     ] START   Staging sources
+[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][76f82637][   build:hello.bst                     ] START   Running commands
+
+    make PREFIX="/usr"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][76f82637][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS integration-commands/hello/76f82637-build.20241106-101653.log
+[00:00:08][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       4
+    Session:     4
+    Fetch Queue: processed 3, skipped 1, failed 0 
+    Build Queue: processed 4, skipped 0, failed 0
+
+

Observe in the build process above, the integration command declared on the +base/alpine.bst element is run after staging the dependency artifacts +into the build sandbox and before running any of the build commands, for +both of the libhello.bst and hello.bst elements.

+

BuildStream assumes that commands which are to be run in the build sandbox +need to be run in an integrated sandbox.

+
+

Tip

+

Integration commands can be taxing on your overall build process, +because they need to run at the beginning of every build which +runtime depends on the element +declaring them.

+

For this reason, it is better to leave out more onerous tasks +if they are not needed at software build time, and handle those +specific tasks differently later in the pipeline, before deployment.

+
+
+
+

4.3.2. Run the hello world program

+

Unlike the previous chapters, this hello world program takes an argument, +we can invoke the program using bst shell:

+ +
+user@host:~/integration-commands$ bst shell hello.bst -- hello pony
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888214.487641    3081 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][76f82637][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][76f82637][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][76f82637][    main:hello.bst                     ] START   Integrating sandbox
+[--:--:--][        ][    main:hello.bst                     ] START   Running commands
+
+    ldconfig "/usr/lib"
+
++ sh -e -c ldconfig "/usr/lib"
+[00:00:00][        ][    main:hello.bst                     ] SUCCESS Running commands
+[00:00:00][76f82637][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][76f82637][    main:hello.bst                     ] STATUS  Running command
+
+    hello pony
+
+Hello pony
+
+

Here we see again, the integration commands are also used when preparing +the shell to launch a command.

+
+
+
+

4.4. Summary

+

In this chapter we’ve observed how integration commands +work, and we now know about public data, which plugins +can read from their dependencies in order to influence their build process.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/tutorial/running-commands.html b/tutorial/running-commands.html new file mode 100644 index 000000000..6cb316897 --- /dev/null +++ b/tutorial/running-commands.html @@ -0,0 +1,543 @@ + + + + + + + + + 2. Running commands — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

2. Running commands

+

In the first chapter we only imported +a file to create an artifact, this time lets run some commands inside +the isolated build sandbox.

+
+

Note

+

This example is distributed with BuildStream +in the doc/examples/running-commands +subdirectory.

+
+
+

2.1. Overview

+

In this chapter, we’ll be running commands inside the sandboxed +execution environment and producing build output.

+

We’ll be compiling the following simple C file:

+
+

2.1.1. files/src/hello.c

+
/*
+ * hello.c - Simple hello world program
+ */
+#include <stdio.h>
+
+int main(int argc, char *argv[])
+{
+  printf("Hello World\n");
+  return 0;
+}
+
+
+

And we’re going to build it using make, using the following Makefile:

+
+
+

2.1.2. files/src/Makefile

+
# Sample makefile for hello.c
+#
+.PHONY: all install
+
+all: hello
+
+install:
+	install -d ${DESTDIR}${PREFIX}/bin
+	install -m 755 hello ${DESTDIR}${PREFIX}/bin
+
+hello: hello.c
+	$(CC) -Wall -o $@ $<
+
+
+

We’ll be using the most fundamental build element, +the manual build element.

+

The manual element is the backbone on which all the other +build elements are built, so understanding how it works at this level is helpful.

+
+
+
+

2.2. Project structure

+

In this project we have a project.conf, a directory with some source +code, and 3 element declarations.

+

Let’s first take a peek at what we need to build using bst show:

+ +
+user@host:~/running-commands$ bst show hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888261.507356    6831 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+fetch needed b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst 
+     waiting bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst 
+     waiting 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst 
+
+

This time we have loaded a pipeline with 3 elements, let’s go over what they do +in detail.

+
+

2.2.1. project.conf

+
# Unique project name
+name: running-commands
+
+# Minimum required BuildStream version
+min-version: 2.0
+
+# Subdirectory where elements are stored
+element-path: elements
+
+# Define an alias for our alpine tarball
+aliases:
+  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
+
+
+

Our project.conf is very much like the last one, except that we +have defined a source alias for alpine.

+
+

Tip

+

Using source aliases for groups of sources +which are generally hosted together is encouraged. This allows one to globally +change the access scheme or URL for a group of repositories which belong together.

+
+
+
+

2.2.2. elements/base/alpine.bst

+
kind: import
+description: |
+
+    Alpine Linux base runtime
+
+sources:
+- kind: tar
+
+  # This is a post doctored, trimmed down system image
+  # of the Alpine linux distribution.
+  #
+  url: alpine:integration-tests-base.v1.x86_64.tar.xz
+  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639
+
+
+

This import element uses a tar +source to download our Alpine Linux tarball to create our base runtime.

+

This tarball is a sysroot which provides the C runtime libraries +and some programs - this is what will be providing the programs we’re +going to run in this example.

+
+
+

2.2.3. elements/base.bst

+
kind: stack
+description: Base stack
+
+depends:
+- base/alpine.bst
+
+
+

This is just a symbolic stack element which declares that +anything which depends on it, will implicitly depend on base/alpine.bst.

+

It is typical to use stack elements in places where the implementing logical +software stack could change, but you rather not have your higher level components +carry knowledge about those changing components.

+

Any element which runtime depends on +the base.bst will now be able to execute programs provided by the imported +base/alpine.bst runtime.

+
+
+

2.2.4. elements/hello.bst

+
kind: manual
+description: |
+
+  Building manually
+
+# Depend on the base system
+depends:
+- base.bst
+
+# Stage the files/src directory for building
+sources:
+  - kind: local
+    path: files/src
+
+# Now configure the commands to run
+config:
+
+  build-commands:
+  - make PREFIX="%{prefix}"
+
+  install-commands:
+  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
+
+
+

Finally we have the element which executes commands. Looking at the +manual element’s documentation, we can see that +the element configuration exposes four command lists:

+
    +

  • configure-commands

    +

    Commands which are run in preparation of a build. This is where you +would normally call any configure stage build tools to configure +the build how you like and generate some files needed for the build.

    +
    • +

  • build-commands

    +

    Commands to run the build, usually a build system will +invoke the compiler for you here.

    +
    • +

  • install-commands

    +

    Commands to install the build results.

    +

    Commands to install the build results into the target system, +these should install files somewhere under %{install-root}.

    +
    • +

  • strip-commands

    +

    Commands to doctor the build results after the install.

    +

    Typically this involves stripping binaries of debugging +symbols or stripping timestamps from build results to ensure +reproducibility.

    +
    • +
+
+

Tip

+

All other build elements +implement exactly the same command lists too, except that they provide +default commands specific to invoke the build systems they support.

+
+

The manual element however is the most basic +and does not provide any default commands, so we have instructed it +to use make to build and install our program.

+
+
+
+

2.3. Using the project

+
+

2.3.1. Build the hello.bst element

+

To build the project, run bst build in the +following way:

+ +
+user@host:~/running-commands$ bst build hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Build
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888261.919086    6859 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+
+BuildStream Version 2.3.0+26.g65e2074ad
+    Session Start: Wednesday, 06-11-2024 at 10:17:41
+    Project:       running-commands (/home/user/buildstream/doc/examples/running-commands)
+    Targets:       hello.bst
+
+User Configuration
+    Configuration File:      /home/user/buildstream/doc/run-bst-drp0a3w0/buildstream.conf
+    Cache Directory:         /home/user/buildstream/doc/run-bst-drp0a3w0
+    Log Files:               /home/user/buildstream/doc/run-bst-drp0a3w0/logs
+    Source Mirrors:          /home/user/buildstream/doc/run-bst-drp0a3w0/sources
+    Build Area:              /home/user/buildstream/doc/run-bst-drp0a3w0/build
+    Strict Build Plan:       Yes
+    Maximum Fetch Tasks:     10
+    Maximum Build Tasks:     4
+    Maximum Push Tasks:      4
+    Maximum Network Retries: 2
+
+Project: running-commands
+
+    Element Plugins
+        manual: core plugin
+        stack:  core plugin
+        import: core plugin
+
+    Source Plugins
+        local: core plugin
+        tar:   core plugin
+
+Pipeline
+fetch needed b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst 
+     waiting bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst 
+     waiting 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst 
+===============================================================================
+[--:--:--][b59f4520][   fetch:base/alpine.bst               ] START   running-commands/base-alpine/b59f4520-fetch.20241106-101741.log
+[--:--:--][b59f4520][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[--:--:--][bb393e5a][   fetch:base.bst                      ] START   running-commands/base/bb393e5a-fetch.20241106-101741.log
+[00:00:00][bb393e5a][   fetch:base.bst                      ] SUCCESS running-commands/base/bb393e5a-fetch.20241106-101741.log
+[00:00:00][b59f4520][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
+[00:00:06][b59f4520][   fetch:base/alpine.bst               ] SUCCESS running-commands/base-alpine/b59f4520-fetch.20241106-101741.log
+[--:--:--][b59f4520][   build:base/alpine.bst               ] START   running-commands/base-alpine/b59f4520-build.20241106-101748.log
+[--:--:--][b59f4520][   build:base/alpine.bst               ] START   Staging sources
+[00:00:00][b59f4520][   build:base/alpine.bst               ] SUCCESS Staging sources
+[--:--:--][b59f4520][   build:base/alpine.bst               ] START   Caching artifact
+[00:00:00][b59f4520][   build:base/alpine.bst               ] SUCCESS Caching artifact
+[00:00:00][b59f4520][   build:base/alpine.bst               ] SUCCESS running-commands/base-alpine/b59f4520-build.20241106-101748.log
+[--:--:--][bb393e5a][   build:base.bst                      ] START   running-commands/base/bb393e5a-build.20241106-101748.log
+[--:--:--][bb393e5a][   build:base.bst                      ] START   Caching artifact
+[00:00:00][bb393e5a][   build:base.bst                      ] SUCCESS Caching artifact
+[00:00:00][bb393e5a][   build:base.bst                      ] SUCCESS running-commands/base/bb393e5a-build.20241106-101748.log
+[--:--:--][8cebf405][   build:hello.bst                     ] START   running-commands/hello/8cebf405-build.20241106-101748.log
+[--:--:--][8cebf405][   build:hello.bst                     ] START   Staging dependencies at: /
+[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
+[--:--:--][8cebf405][   build:hello.bst                     ] START   Integrating sandbox
+[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][8cebf405][   build:hello.bst                     ] START   Staging sources
+[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Staging sources
+[--:--:--][8cebf405][   build:hello.bst                     ] START   Running commands
+
+    make PREFIX="/usr"
+    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
+
+[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Running commands
+[--:--:--][8cebf405][   build:hello.bst                     ] START   Caching artifact
+[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Caching artifact
+[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS running-commands/hello/8cebf405-build.20241106-101748.log
+[00:00:06][        ][    main:core activity                 ] SUCCESS Build
+
+Pipeline Summary
+    Total:       3
+    Session:     3
+    Fetch Queue: processed 2, skipped 1, failed 0 
+    Build Queue: processed 3, skipped 0, failed 0
+
+

Now we’ve built our hello world program, using make +and the C compiler provided by the Alpine Linux image.

+

In the first chapter we observed that the inputs +and output of an element are directory trees. In this example, the directory tree +generated by base/alpine.bst is consumed by hello.bst due to the +implicit runtime dependency introduced by base.bst.

+
+

Tip

+

All of the dependencies which are required to run for +the sake of a build, are staged at the root of the build sandbox. These comprise the +runtime environment in which the depending element will run commands.

+

The result is that the make program and C compiler provided by base/alpine.bst +were already in $PATH and ready to run when the commands were needed by hello.bst.

+
+

Now observe that all of the elements in the loaded pipeline are cached, +the element is built:

+ +
+user@host:~/running-commands$ bst show hello.bst
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888269.262913    6919 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+      cached b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst 
+      cached bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst 
+      cached 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst 
+
+
+
+

2.3.2. Run the hello world program

+

Now that we’ve built everything, we can indulge ourselves in running +the hello world program using bst shell:

+ +
+user@host:~/running-commands$ bst shell hello.bst -- hello
+
+[--:--:--][        ][    main:core activity                 ] START   Loading elements
+[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
+[--:--:--][        ][    main:core activity                 ] START   Resolving elements
+WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
+I0000 00:00:1730888269.683298    6947 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
+[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
+[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
+[--:--:--][        ][    main:core activity                 ] START   Query cache
+[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
+[--:--:--][8cebf405][    main:hello.bst                     ] START   Staging dependencies
+[00:00:00][8cebf405][    main:hello.bst                     ] SUCCESS Staging dependencies
+[--:--:--][8cebf405][    main:hello.bst                     ] START   Integrating sandbox
+[00:00:00][8cebf405][    main:hello.bst                     ] SUCCESS Integrating sandbox
+[--:--:--][8cebf405][    main:hello.bst                     ] STATUS  Running command
+
+    hello
+
+Hello World
+
+

Here, bst shell created a runtime environment for running +the hello.bst element. This was done by staging all of the dependencies of +hello.bst including the hello.bst output itself into a directory. Once a directory +with all of the dependencies was staged and ready, we ran the hello command from +within the build sandbox environment.

+
+

Tip

+

When specifying a command for bst shell to run, +we always specify -- first. This is a commonly understood shell syntax +to indicate that the remaining arguments are to be treated literally.

+

Specifying -- is optional and disambiguates BuildStream’s arguments +and options from those of the program being run by +bst shell.

+
+
+
+
+

2.4. Summary

+

In this chapter we’ve explored how to use the manual element, +which forms the basis of all build elements.

+

We’ve also observed how the directory tree from the output artifact of one element +is later staged at the root of the sandbox, as input for use by any build elements +which depend on that element.

+
+

Tip

+

The way that elements consume their dependency input can vary across the +different kinds of elements. This chapter describes how it works for +build elements implementations, which +are the most commonly used element type.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_commands.html b/using_commands.html new file mode 100644 index 000000000..315a2bbe8 --- /dev/null +++ b/using_commands.html @@ -0,0 +1,1571 @@ + + + + + + + + + Commands — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Commands

+

This page contains documentation for each BuildStream command, +along with their possible options and arguments. Each command can be +invoked on the command line, where, in most cases, this will be from the +project’s main directory.

+
+

Commonly used parameters

+
+

Remotes

+

Remote cache servers can be specified on the +command line for commands which may result in communicating with such servers.

+

Any command which has arguments to specify a REMOTE, such as --artifact-remote +or --source-remote, will override whatever was set in the user configuration, +and will have an accompanying switch which allows the command to decide whether +to ignore any remote artifact or source +caches suggested by project configuration.

+

Remotes can be specified on the command line either as a simple URI, or as +a comma separated list of key value pairs.

+

Specifying a remote using a URI

+
bst artifact push --remote https://artifacts.com/artifacts:8088 element.bst
+
+
+

Specifying a remote using key value pairs

+
bst build --artifact-remote \
+    url=https://artifacts.com/artifacts:8088,type=index,server-cert=~/artifacts.cert \
+    element.bst
+
+
+
+

Attributes

+

Here is the list attributes which can be spefied when providing a REMOTE on the command line:

+
    +

  • url

    +

    The URL of the remote, possibly including a port number.

    +
    • +

  • instance-name

    +

    The instance name of this remote, used for sharding by some implementations.

    +
    • +

  • type

    +

    Whether this remote is to be used for indexing, storage or both, as explained +in the corresponding user configuration documentation

    +
    • +

  • push

    +

    Normally one need not specify this, as it is often inferred by the command +being used. In some cases, like bst build, it can +be useful to specify multiple remotes, and only allow pushing to some of +the remotes.

    +

    If unspecified, this is assumed to be True and BuildStream will attempt +to push to the remote, but fallback to only pulling if insufficient credentials +were provided.

    +
    • +

  • server-cert, client-cert, client-key:

    +

    These keys specify the attributes of the authentication configuration.

    +

    When specifying these on the command line, they are interpreted as paths relative +to the current working directory.

    +
    • +
+
+
+
+
+

Top-level commands

+
+

bst

+

Build and manipulate BuildStream projects

+

Most of the main options override options in the +user preferences configuration file.

+
bst [OPTIONS] COMMAND [ARGS]...
+
+
+

Options

+
+
+--version
+
+ +
+
+-c, --config <config>
+

Configuration file to use

+
+ +
+
+-C, --directory <directory>
+

Project directory (default: current directory)

+
+ +
+
+--on-error <on_error>
+

What to do when an error is encountered

+
+
Options:
+

continue | quit | terminate

+
+
+
+ +
+
+--fetchers <fetchers>
+

Maximum simultaneous download tasks

+
+ +
+
+--builders <builders>
+

Maximum simultaneous build tasks

+
+ +
+
+--pushers <pushers>
+

Maximum simultaneous upload tasks

+
+ +
+
+--max-jobs <max_jobs>
+

Number of parallel jobs allowed for a given build task

+
+ +
+
+--network-retries <network_retries>
+

Maximum retries for network tasks

+
+ +
+
+--no-interactive
+

Force non interactive mode, otherwise this is automatically decided

+
+ +
+
+--verbose, --no-verbose
+

Be extra verbose

+
+ +
+
+--debug, --no-debug
+

Print debugging output

+
+ +
+
+--error-lines <error_lines>
+

Maximum number of lines to show from a task log

+
+ +
+
+--message-lines <message_lines>
+

Maximum number of lines to show in a detailed message

+
+ +
+
+--log-file <log_file>
+

A file to store the main log (allows storing the main log while in interactive mode)

+
+ +
+
+--colors, --no-colors
+

Force enable/disable ANSI color codes in output

+
+ +
+
+--strict, --no-strict
+

Elements must be rebuilt when their dependencies have changed

+
+ +
+
+-o, --option <OPTION VALUE>
+

Specify a project option

+
+ +
+
+--default-mirror <default_mirror>
+

The mirror to fetch from first, before attempting other mirrors

+
+ +
+
+--pull-buildtrees
+

Include an element’s build tree when pulling remote element artifacts

+
+ +
+
+--cache-buildtrees <cache_buildtrees>
+

Cache artifact build tree content on creation

+
+
Options:
+

always | auto | never

+
+
+
+ +

Commands

+
+
+artifact
+

Manipulate cached artifacts.

+
+ +
+
+build
+

Build elements in a pipeline

+
+ +
+
+help
+

Print usage information

+
+ +
+
+init
+

Initialize a new BuildStream project

+
+ +
+
+shell
+

Shell into an element’s sandbox environment

+
+ +
+
+show
+

Show elements in the pipeline

+
+ +
+
+source
+

Manipulate sources for an element

+
+ +
+
+workspace
+

Manipulate developer workspaces

+
+ +
+
+
+

bst artifact

+

Manipulate cached artifacts

+

Some subcommands take artifact references as arguments. Artifacts +can be specified in two ways:

+
+
- artifact refs: triples of the form <project name>/<element name>/<cache key>
+
- element names
+
+

When elements are given, the artifact is looked up by observing the element +and it’s current cache key.

+

The commands also support shell-style wildcard expansion: ? matches a +single character, * matches zero or more characters but does not match the / +path separator, and ** matches zero or more characters including / path separators.

+

If the wildcard expression ends with .bst, then it will be used to search +element names found in the project, otherwise, it will be used to search artifacts +that are present in the local artifact cache.

+

Some example arguments are:

+
+
- myproject/hello/8276376b077eda104c812e6ec2f488c7c9eea211ce572c83d734c10bf241209f
+
- myproject/he*/827637*
+
- core/*.bst (all elements in the core directory)
+
- **.bst (all elements)
+
- myproject/** (all artifacts from myproject)
+
- myproject/myelement/* (all cached artifacts for a specific element)
+
+
bst artifact [OPTIONS] COMMAND [ARGS]...
+
+
+

Commands

+
+
+checkout
+

Checkout contents of an artifact

+
+ +
+
+delete
+

Remove artifacts from the local cache

+
+ +
+
+list-contents
+

List the contents of an artifact

+
+ +
+
+log
+

Show logs of artifacts

+
+ +
+
+pull
+

Pull a built artifact

+
+ +
+
+push
+

Push a built artifact

+
+ +
+
+show
+

Show the cached state of artifacts

+
+ +
+
+
+

bst init

+

Initialize a new BuildStream project

+

Creates a new BuildStream project.conf in the project +directory.

+

Unless –project-name is specified, this will be an +interactive session.

+
bst init [OPTIONS] [TARGET_DIRECTORY]
+
+
+

Options

+
+
+--project-name <project_name>
+

The project name to use

+
+ +
+
+--min-version <min_version>
+

The required format version

+
+
Default:
+

'2.3'

+
+
+
+ +
+
+--element-path <element_path>
+

The subdirectory to store elements in

+
+
Default:
+

'elements'

+
+
+
+ +
+
+-f, --force
+

Allow overwriting an existing project.conf

+
+ +

Arguments

+
+
+TARGET_DIRECTORY
+

Optional argument

+
+ +
+
+
+

bst build

+

Build elements in a pipeline

+

Specifying no elements will result in building the default targets +of the project. If no default targets are configured, all project +elements will be built.

+

When this command is executed from a workspace directory, the default +is to build the workspace element.

+

Specify –deps to control which dependencies must be built:

+
+
none: No dependencies, just the element itself
+
build: Build time dependencies, excluding the element itself
+
all: All dependencies
+
+

Dependencies that are consequently required to build the requested +elements will be built on demand.

+
bst build [OPTIONS] [ELEMENTS]...
+
+
+

Options

+
+
+-d, --deps <deps>
+

The dependencies to build

+
+
Options:
+

none | build | all

+
+
+
+ +
+
+--artifact-remote <artifact_remotes>
+

A remote for uploading and downloading artifacts

+
+ +
+
+--source-remote <source_remotes>
+

A remote for uploading and downloading cached sources

+
+ +
+
+--ignore-project-artifact-remotes
+

Ignore remote artifact cache servers recommended by projects

+
+ +
+
+--ignore-project-source-remotes
+

Ignore remote source cache servers recommended by projects

+
+ +
+
+-r, --retry-failed
+

Try to build elements for which a failed build artifact is found

+
+ +

Arguments

+
+
+ELEMENTS
+

Optional argument(s)

+
+ +
+
+
+

bst show

+

Show elements in the pipeline

+

Specifying no elements will result in showing the default targets +of the project. If no default targets are configured, all project +elements will be shown.

+

When this command is executed from a workspace directory, the default +is to show the workspace element.

+

By default this will show all of the dependencies of the +specified target element.

+

Specify --deps to control which elements to show:

+
+
none: No dependencies, just the element itself
+
run: Runtime dependencies, including the element itself
+
build: Build time dependencies, excluding the element itself
+
all: All dependencies
+
+

FORMAT

+

The --format option controls what should be printed for each element, +the following symbols can be used in the format string:

+
+
%{name} The element name
+
%{description} The element description, on a single line (Since: 2.3)
+
%{key} The abbreviated cache key (if all sources are consistent)
+
%{full-key} The full cache key (if all sources are consistent)
+
%{state} cached, buildable, waiting, inconsistent or junction
+
%{config} The element configuration
+
%{vars} Variable configuration
+
%{env} Environment settings
+
%{public} Public domain data
+
%{workspaced} If the element is workspaced
+
%{workspace-dirs} A list of workspace directories
+
%{deps} A list of all dependencies
+
%{build-deps} A list of build dependencies
+
%{runtime-deps} A list of runtime dependencies
+
+

The value of the %{symbol} without the leading ‘%’ character is understood +as a pythonic formatting string, so python formatting features apply, +example:

+
+
bst show target.bst –format
+
+
‘Name: %{name: ^20} Key: %{key: ^8} State: %{state}’
+
+
+

If you want to use a newline in a format string in bash, use the ‘$’ modifier:

+
+
bst show target.bst –format
+
+
$’———- %{name} ———-n%{vars}’
+
+
+
bst show [OPTIONS] [ELEMENTS]...
+
+
+

Options

+
+
+--except <except_>
+

Except certain dependencies

+
+ +
+
+-d, --deps <deps>
+

The dependencies to show

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac69c0>

+
+
Options:
+

none | run | build | all

+
+
+
+ +
+
+--order <order>
+

Staging or alphabetic ordering of dependencies

+
+
Default:
+

'stage'

+
+
Options:
+

stage | alpha

+
+
+
+ +
+
+-f, --format <FORMAT>
+

Format string for each element

+
+ +

Arguments

+
+
+ELEMENTS
+

Optional argument(s)

+
+ +
+
+
+

bst shell

+

Run a command in the target element’s sandbox environment

+

When this command is executed from a workspace directory, the default +is to shell into the workspace element.

+

This will stage a temporary sysroot for running the target +element, assuming it has already been built and all required +artifacts are in the local cache.

+

Use ‘–’ to separate a command from the options to bst, +otherwise bst may respond to them instead. e.g.

+
+
bst shell example.bst – df -h
+
+

Use the –build option to create a temporary sysroot for +building the element instead.

+

If no COMMAND is specified, the default is to attempt +to run an interactive shell.

+
bst shell [OPTIONS] [TARGET] [COMMAND]...
+
+
+

Options

+
+
+-b, --build
+

Stage dependencies and sources to build

+
+ +
+
+--mount <HOSTPATH PATH>
+

Mount a file or directory into the sandbox

+
+ +
+
+--isolate
+

Create an isolated build sandbox

+
+ +
+
+-t, --use-buildtree
+

Stage a buildtree. Will fail if a buildtree is not available. pull-buildtrees configuration is needed if the buildtree is not available locally.

+
+ +
+
+--artifact-remote <artifact_remotes>
+

A remote for uploading and downloading artifacts

+
+ +
+
+--source-remote <source_remotes>
+

A remote for uploading and downloading cached sources

+
+ +
+
+--ignore-project-artifact-remotes
+

Ignore remote artifact cache servers recommended by projects

+
+ +
+
+--ignore-project-source-remotes
+

Ignore remote source cache servers recommended by projects

+
+ +

Arguments

+
+
+TARGET
+

Optional argument

+
+ +
+
+COMMAND
+

Optional argument(s)

+
+ +
+
+
+

bst source

+

Manipulate sources for an element

+
bst source [OPTIONS] COMMAND [ARGS]...
+
+
+

Commands

+
+
+checkout
+

Checkout sources of an element

+
+ +
+
+fetch
+

Fetch sources in a pipeline

+
+ +
+
+push
+

Push sources in a pipeline

+
+ +
+
+track
+

Track new source references

+
+ +
+
+
+

bst workspace

+

Manipulate developer workspaces

+
bst workspace [OPTIONS] COMMAND [ARGS]...
+
+
+

Commands

+
+
+close
+

Close workspaces

+
+ +
+
+list
+

List open workspaces

+
+ +
+
+open
+

Open a new workspace

+
+ +
+
+reset
+

Reset a workspace to its original state

+
+ +
+
+
+

Artifact subcommands

+
+

Artifact names

+

Various artifact subcommands accept either element names, +which will operate on artifacts by deriving the artifact from local project state, +or artifact names interchangeably as targets. Artifact names allow +the user to operate directly on cached artifacts, without requiring local project data.

+

An artifact name is composed of the following identifiers:

+
    +

  • The project name

    • +

  • The element name, without any trailing .bst extension

    • +

  • The cache key of the element at the time it was built.

    • +
+

To compose an artifact name, simply join these using a forward slash (/) character, like so: <project-name>/<element-name>/<cache-key>.

+

An artifact name might look like: project/target/788da21e7c1b5818b7e7b60f7eb75841057ff7e45d362cc223336c606fe47f27

+
+

bst artifact checkout

+

Checkout contents of an artifact

+

When this command is executed from a workspace directory, the default +is to checkout the artifact of the workspace element.

+
bst artifact checkout [OPTIONS] [TARGET]
+
+
+

Options

+
+
+-f, --force
+

Allow files to be overwritten

+
+ +
+
+-d, --deps <deps>
+

The dependencies to checkout

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6a20>

+
+
Options:
+

run | build | none | all

+
+
+
+ +
+
+--integrate, --no-integrate
+

Whether to run integration commands

+
+ +
+ +

Checkout hardlinks instead of copying if possible

+
+ +
+
+--tar <LOCATION>
+

Create a tarball from the artifact contents instead of a file tree. If LOCATION is ‘-’, the tarball will be dumped to the standard output.

+
+ +
+
+--compression <compression>
+

The compression option of the tarball created.

+
+
Options:
+

gz | xz | bz2

+
+
+
+ +
+
+--directory <directory>
+

The directory to checkout the artifact to

+
+ +
+
+--artifact-remote <artifact_remotes>
+

A remote for downloading artifacts

+
+ +
+
+--ignore-project-artifact-remotes
+

Ignore remote artifact cache servers recommended by projects

+
+ +

Arguments

+
+
+TARGET
+

Optional argument

+
+ +
+
+
+

bst artifact log

+

Show build logs of artifacts

+
bst artifact log [OPTIONS] [ARTIFACTS]...
+
+
+

Options

+
+
+--out <out>
+

Output logs to individual files in the specified path. If absent, logs are written to stdout.

+
+ +

Arguments

+
+
+ARTIFACTS
+

Optional argument(s)

+
+ +
+
+
+

bst artifact pull

+

Pull a built artifact from the configured remote artifact cache.

+

Specifying no elements will result in pulling the default targets +of the project. If no default targets are configured, all project +elements will be pulled.

+

When this command is executed from a workspace directory, the default +is to pull the workspace element.

+

By default the artifact will be pulled one of the configured caches +if possible, following the usual priority order. If the –artifact-remote +flag is given, only the specified cache will be queried.

+

Specify –deps to control which artifacts to pull:

+
+
none: No dependencies, just the element itself
+
run: Runtime dependencies, including the element itself
+
build: Build time dependencies, excluding the element itself
+
all: All dependencies
+
+
bst artifact pull [OPTIONS] [ARTIFACTS]...
+
+
+

Options

+
+
+-d, --deps <deps>
+

The dependency artifacts to pull

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6960>

+
+
Options:
+

build | none | run | all

+
+
+
+ +
+
+--artifact-remote <artifact_remotes>
+

A remote for downloading artifacts

+
+ +
+
+--ignore-project-artifact-remotes
+

Ignore remote artifact cache servers recommended by projects

+
+ +

Arguments

+
+
+ARTIFACTS
+

Optional argument(s)

+
+ +
+
+
+

bst artifact push

+

Push built artifacts to a remote artifact cache, possibly pulling them first.

+

Specifying no elements will result in pushing the default targets +of the project. If no default targets are configured, all project +elements will be pushed.

+

When this command is executed from a workspace directory, the default +is to push the workspace element.

+

If bst has been configured to include build trees on artifact pulls, +an attempt will be made to pull any required build trees to avoid the +skipping of partial artifacts being pushed.

+

Specify –deps to control which artifacts to push:

+
+
none: No dependencies, just the element itself
+
run: Runtime dependencies, including the element itself
+
build: Build time dependencies, excluding the element itself
+
all: All dependencies
+
+
bst artifact push [OPTIONS] [ARTIFACTS]...
+
+
+

Options

+
+
+-d, --deps <deps>
+

The dependencies to push

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6960>

+
+
Options:
+

build | none | run | all

+
+
+
+ +
+
+--artifact-remote <artifact_remotes>
+

A remote for uploading artifacts

+
+ +
+
+--ignore-project-artifact-remotes
+

Ignore remote artifact cache servers recommended by projects

+
+ +

Arguments

+
+
+ARTIFACTS
+

Optional argument(s)

+
+ +
+
+
+

bst artifact delete

+

Remove artifacts from the local cache

+
bst artifact delete [OPTIONS] [ARTIFACTS]...
+
+
+

Options

+
+
+-d, --deps <deps>
+

The dependencies to delete

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6960>

+
+
Options:
+

none | run | build | all

+
+
+
+ +

Arguments

+
+
+ARTIFACTS
+

Optional argument(s)

+
+ +
+
+
+

bst artifact show

+

show the cached state of artifacts

+
bst artifact show [OPTIONS] [ARTIFACTS]...
+
+
+

Options

+
+
+-d, --deps <deps>
+

The dependencies we also want to show

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6960>

+
+
Options:
+

build | run | all | none

+
+
+
+ +

Arguments

+
+
+ARTIFACTS
+

Optional argument(s)

+
+ +
+
+
+

bst artifact list-contents

+

List the contents of an artifact.

+

Note that ‘artifacts’ can be element names, which must end in ‘.bst’, +or artifact references, which must be in the format <project_name>/<element>/<key>.

+
bst artifact list-contents [OPTIONS] [ARTIFACTS]...
+
+
+

Options

+
+
+-l, --long
+

Provide more information about the contents of the artifact.

+
+ +

Arguments

+
+
+ARTIFACTS
+

Optional argument(s)

+
+ +
+
+
+
+

Source subcommands

+
+

bst source fetch

+

Fetch sources required to build the pipeline

+

Specifying no elements will result in fetching the default targets +of the project. If no default targets are configured, all project +elements will be fetched.

+

When this command is executed from a workspace directory, the default +is to fetch the workspace element.

+

By default this will only try to fetch sources for the specified +elements.

+

Specify –deps to control which sources to fetch:

+
+
none: No dependencies, just the element itself
+
run: Runtime dependencies, including the element itself
+
build: Build time dependencies, excluding the element itself
+
all: All dependencies
+
+
bst source fetch [OPTIONS] [ELEMENTS]...
+
+
+

Options

+
+
+--except <except_>
+

Except certain dependencies from fetching

+
+ +
+
+-d, --deps <deps>
+

The dependencies to fetch

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6960>

+
+
Options:
+

none | build | run | all

+
+
+
+ +
+
+--source-remote <source_remotes>
+

A remote for downloading sources

+
+ +
+
+--ignore-project-source-remotes
+

Ignore remote source cache servers recommended by projects

+
+ +

Arguments

+
+
+ELEMENTS
+

Optional argument(s)

+
+ +
+
+
+

bst source track

+

Consults the specified tracking branches for new versions available +to build and updates the project with any newly available references.

+

Specifying no elements will result in tracking the default targets +of the project. If no default targets are configured, all project +elements will be tracked.

+

When this command is executed from a workspace directory, the default +is to track the workspace element.

+

If no default is declared, all elements in the project will be tracked

+

By default this will track just the specified element, but you can also +update a whole tree of dependencies in one go.

+

Specify –deps to control which sources to track:

+
+
none: No dependencies, just the specified elements
+
all: All dependencies of all specified elements
+
+
bst source track [OPTIONS] [ELEMENTS]...
+
+
+

Options

+
+
+--except <except_>
+

Except certain dependencies from tracking

+
+ +
+
+-d, --deps <deps>
+

The dependencies to track

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6960>

+
+
Options:
+

build | run | all | none

+
+
+
+ +
+
+-J, --cross-junctions
+

Allow crossing junction boundaries

+
+ +

Arguments

+
+
+ELEMENTS
+

Optional argument(s)

+
+ +
+
+
+

bst source push

+

Push sources required to build the pipeline

+

Specifying no elements will result in pushing the sources of the default +targets of the project. If no default targets are configured, sources of +all project elements will be pushed.

+

When this command is executed from a workspace directory, the default +is to push the sources of the workspace element.

+

By default this will only try to push sources for the specified +elements.

+

Specify –deps to control which sources to fetch:

+
+
none: No dependencies, just the element itself
+
run: Runtime dependencies, including the element itself
+
build: Build time dependencies, excluding the element itself
+
all: All dependencies
+
+
bst source push [OPTIONS] [ELEMENTS]...
+
+
+

Options

+
+
+--except <except_>
+

Except certain dependencies from pushing

+
+ +
+
+-d, --deps <deps>
+

The dependencies to push

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6960>

+
+
Options:
+

none | build | run | all

+
+
+
+ +
+
+--source-remote <source_remotes>
+

A remote for uploading sources

+
+ +
+
+--ignore-project-source-remotes
+

Ignore remote source cache servers recommended by projects

+
+ +

Arguments

+
+
+ELEMENTS
+

Optional argument(s)

+
+ +
+
+
+

bst source checkout

+

Checkout sources of an element to the specified location

+

When this command is executed from a workspace directory, the default +is to checkout the sources of the workspace element.

+
bst source checkout [OPTIONS] [ELEMENT]
+
+
+

Options

+
+
+-f, --force
+

Allow files to be overwritten

+
+ +
+
+--except <except_>
+

Except certain dependencies

+
+ +
+
+-d, --deps <deps>
+

The dependencies whose sources to checkout

+
+
Default:
+

<buildstream.types._PipelineSelection object at 0x7f4731ac6960>

+
+
Options:
+

build | none | run | all

+
+
+
+ +
+
+--tar <LOCATION>
+

Create a tarball containing the sources instead of a file tree.

+
+ +
+
+--compression <compression>
+

The compression option of the tarball created.

+
+
Options:
+

gz | xz | bz2

+
+
+
+ +
+
+--include-build-scripts
+
+ +
+
+--directory <directory>
+

The directory to checkout the sources to

+
+ +
+
+--source-remote <source_remotes>
+

A remote for downloading cached sources

+
+ +
+
+--ignore-project-source-remotes
+

Ignore remote source cache servers recommended by projects

+
+ +

Arguments

+
+
+ELEMENT
+

Optional argument

+
+ +
+
+
+

Workspace subcommands

+
+

bst workspace open

+

Open a workspace for manual source modification

+
bst workspace open [OPTIONS] ELEMENTS...
+
+
+

Options

+
+
+--no-checkout
+

Do not checkout the source, only link to the given directory

+
+ +
+
+-f, --force
+

The workspace will be created even if the directory in which it will be created is not empty or if a workspace for that element already exists

+
+ +
+
+--directory <directory>
+

Only for use when a single Element is given: Set the directory to use to create the workspace

+
+ +
+
+--source-remote <source_remotes>
+

A remote for downloading cached sources

+
+ +
+
+--ignore-project-source-remotes
+

Ignore remote source cache servers recommended by projects

+
+ +

Arguments

+
+
+ELEMENTS
+

Required argument(s)

+
+ +
+
+
+

bst workspace close

+

Close a workspace

+
bst workspace close [OPTIONS] [ELEMENTS]...
+
+
+

Options

+
+
+--remove-dir
+

Remove the path that contains the closed workspace

+
+ +
+
+-a, --all
+

Close all open workspaces

+
+ +

Arguments

+
+
+ELEMENTS
+

Optional argument(s)

+
+ +
+
+
+

bst workspace reset

+

Reset a workspace to its original state

+
bst workspace reset [OPTIONS] [ELEMENTS]...
+
+
+

Options

+
+
+--soft
+

Mark workspace to re-execute configuration steps (if any) on next build. Does not alter workspace contents.

+
+ +
+
+-a, --all
+

Reset all open workspaces

+
+ +

Arguments

+
+
+ELEMENTS
+

Optional argument(s)

+
+ +
+
+
+

bst workspace list

+

List open workspaces

+
bst workspace list [OPTIONS]
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_config.html b/using_config.html new file mode 100644 index 000000000..281d36c95 --- /dev/null +++ b/using_config.html @@ -0,0 +1,1433 @@ + + + + + + + + + User configuration — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

User configuration

+

User configuration and preferences can be specified in a user provided +configuration file, and in most cases these preferences can be overridden +using command line options.

+

Values that are not specified in a user configuration file assume +their default values.

+
+

Configuration file

+

Unless a configuration file is explicitly specified on the command line +when invoking bst, an attempt is made to load user specific configuration from +$XDG_CONFIG_HOME/buildstream.conf. On most Linux based systems, the location +will be ~/.config/buildstream.conf

+
+

Note

+

If you have have multiple major versions of BuildStream installed, you +can have separate configuration files in your ${XDG_CONFIG_HOME}.

+

You can do this by naming them according to the major versions of +BuildStream you have installed. BuildStream 1 will load its configuration +from $XDG_CONFIG_HOME/buildstream1.conf and BuildStream 2 will load +its configuration from $XDG_CONFIG_HOME/buildstream2.conf, while +any version will fallback to $XDG_CONFIG_HOME/buildstream.conf.

+
+
+
+

Working directories

+

The working directories control where BuildStream will store data. While +these will have sane defaults, you may want to change these directories +depending on your partitioning scheme and where you may have extra space.

+

Environment variables and ~/ home directory expressions are supported +when specifying working directories.

+

Working directories are configured at the toplevel of your configuration file, like so:

+
#
+# Configure working directories
+#
+sourcedir: ~/buildstream/sources
+
+
+
+

Attributes

+
    +

  • sourcedir

    +

    This is the location where BuildStream stores the source code it downloads +for builds.

    +
    • +

  • logdir

    +

    This is the location where BuildStream stores log files of build command +output and basically logs pertaining to any activity BuildStream orchestrates.

    +
    • +

  • cachedir

    +

    This is the location where BuildStream stores the local CAS (Content Addressable Storage).

    +

    The CAS is used to cache anything and everything which BuildStream may +reuse at a later time.

    +
    +

    Attention

    +

    While it may be beneficial at times to delete the entire cache directory +manually in order to free up disk space, one should keep in mind that +the sourcedir and logdir are configured as subdirectories of +this directory when default configuration is used.

    +

    Take care not to accidentally remove all your cached downloaded sources +when deleting your cache.

    +
    +
    • +

  • workspacedir

    +

    A default location for opening workspaces.

    +
    +

    Note

    +

    By default this is configured to ., which is to say workspaces are +created as a subdirectory of the current working directory by default.

    +

    Because of this, the workspacedir directory is the only directory +which is allowed to be declared as a relative path.

    +
    +
    • +
+
+
+
+

Cache control

+

Beyond deciding what directory you intend to place the cache, there are +some controls on what is cached locally and how.

+

These are controlled by the attributes of a cache dictionary at the +toplevel of your configuration file, like so:

+
#
+# Control the cache
+#
+cache:
+
+  # Use as much space as possible
+  quota: infinity
+
+  # Keep 5% of disk space available
+  reserved-disk-space: 5%
+
+  # Retain 80% of the cache on cleanup
+  low-watermark: 80%
+
+  # Avoid pulling large amounts of data we don't need locally
+  pull-buildtrees: False
+
+  #
+  # Avoid caching build trees if we don't need them
+  cache-buildtrees: auto
+
+  # Connection config is parameters given to grpc. It's completely
+  # optional. By default keepalive time is unset and grpc defaults
+  # are used.
+  connection-config:
+    keepalive-time: 60
+
+  #
+  # Support CAS server as remote cache
+  # Useful to minimize network traffic with remote execution
+  # or to work with limited local disk space
+  storage-service:
+    url: https://cache-server.com/cas:11001
+    auth:
+      server-cert: server.crt
+      client-cert: client.crt
+      client-key: client.key
+      access-token: access.token
+      access-token-reload-interval: 60
+
+
+
+

Attributes

+
    +

  • quota

    +

    This controls how much data you allow BuildStream to cache locally.

    +

    An attempt will be made to error out instead of exceeding the maximum +quota which the user has allowed here. Given that it is impossible for +BuildStream to know how much data a given build will create, this quota +is implemented on a best effort basis.

    +

    The quota can be specified in multiple ways:

    +
      +

    • The special infinity value

      +

      This default value states that BuildStream can use as much space as +is available on the filesystem where the cache resides.

      +
      • +

    • A number in bytes.

      • +

    • A human readable number, suffixed in K, M, G or T

      +

      E.g. 250K being 250 kilobytes, 100M being 100 megabytes, etc.

      +
      • +

    • A percentage value, e.g. 80%

      +

      Percentage values are taken to represent a percentage of the partition +size on the filesystem where the cache has been configured.

      +
      • +
    +
    • +

  • reserved-disk-space

    +

    This controls how much disk space should remain available. If the amount +of available disk space falls below the specified value, unused cache +objects will be pruned even if the configured quota has not been exceeded.

    +

    reserved-disk-space can be specified in the same way as quota, with +the exception of the special infinity value. The default is 5%.

    +
    • +

  • low-watermark

    +

    This controls how much of the cache should be retained on cleanup.

    +

    low-watermark is specified as a percentage of the effective cache quota +as configured by quota and/or reserved-disk-space. The default is +80%, which means that when cleanup is triggered, 20% of the cache will +be pruned by removing CAS objects that haven’t been used recently.

    +
    • +

  • pull-buildtrees

    +

    Whether to pull build trees when downloading remote artifacts.

    +

    The build tree of an artifact is the directory where a build took +place, this is useful for running a build shell +in order to observe how an element was built or to debug how a +build failed if the build failed remotely.

    +

    Since build trees are rather expensive, the default is to not pull +build trees for every artifact. If you need a build tree that exists +remotely, it will be possible to download it as an option at the +time you run a command which requires it.

    +
    • +

  • cache-buildtrees

    +

    Whether to cache build trees when creating artifacts, if build trees +are cached locally and the client is configured to push to remote servers, +then build trees will be pushed along with any uploaded artifacts.

    +

    This configuration has three possible values:

    +
      +

    • never: Never cache build trees

      • +

    • auto: Only cache the build trees where necessary (e.g. for failed builds)

      • +

    • always: Always cache the build tree.

      • +
    +
    • +

  • storage-service

    +

    An optional service configuration +to use a Content Addressable Storage service as a remote cache. Write access +is required.

    +

    This service is compatible with the storage service offered by +cache servers.

    +

    Without this option, all content is stored in the local cache. This includes +CAS objects from fetched sources, build outputs and pulled artifacts. +With this option, content is primarily stored in the remote cache and the +local cache is populated only as needed. E.g. bst artifact checkout +will download CAS objects on demand from the remote cache. +This feature is incompatible with offline operation.

    +

    This is primarily useful in combination with +remote execution to minimize downloads +of build outputs, which may not be needed locally. The elimination of +unnecessary downloads reduces total build time, especially if the bandwidth +between the local system and the remote execution cluster is limited.

    +
    +

    Tip

    +

    Skip the storage-service option in the +remote execution configuration to +use the same CAS service for caching and remote execution.

    +
    +

    It is also possible to configure this with local builds without remote +execution. This enables operation with a small local cache even with large +projects. However, for local builds this can result in a longer total build +time due to additional network transfers. This is only recommended with a +high bandwidth connection to a storage-service, ideally in a local network.

    +
    • +
+
+
+
+

Scheduler controls

+

Controls related to how the scheduler works are exposed as attributes of the +toplevel scheduler dictionary, like so:

+
#
+# Control the scheduler
+#
+scheduler:
+
+  # Allow building up to four seperate elements at a time
+  builders: 4
+
+  # Continue building as many elements as possible if anything fails
+  on-error: continue
+
+
+
+

Attributes

+
    +

  • fetchers

    +

    The number of concurrent tasks which download sources or artifacts.

    +
    • +

  • pushers

    +

    The number of concurrent tasks which upload sources or artifacts.

    +
    • +

  • builders

    +

    The number of concurrent tasks which build elements.

    +
    +

    Note

    +

    This does not control the number of processes in the scope of the +build of a single element, but rather the number of elements which +may be built in parallel.

    +
    +
    • +

  • network-retries

    +

    The number of times to retry a task which failed due to network connectivity issues.

    +
    • +

  • on-error

    +

    What to do when a task fails and BuildStream is running in non-interactive mode. This can +be set to the following values:

    +
      +

    • continue: Continue with other tasks, a summary of errors will be printed at the end

      • +

    • quit: Quit after all ongoing tasks have completed

      • +

    • terminate: Abort any ongoing tasks and exit immediately

      • +
    +
    +

    Note

    +

    If BuildStream is running in interactive mode, then the ongoing build will be suspended +and the user will be prompted and asked what to do when a task fails.

    +

    Interactive mode is automatically enabled if BuildStream is connected to a terminal +rather than being run automatically, or, it can be specified on the command line.

    +
    +
    • +
+
+
+
+

Build controls

+

Some aspects about how elements get built can be controlled by attributes of the build +dictionary at the toplevel, like so:

+
#
+# Build controls
+#
+build:
+
+  #
+  # Allow up to 4 parallel processes to execute within the scope of one build
+  #
+  max-jobs: 4
+
+
+
+

Attributes

+
    +

  • max-jobs

    +

    This is a best effort attempt to instruct build systems on how many parallel +processes to use when building an element.

    +

    It is supported by most popular build systems such as make, cmake, ninja, +etc, via environment variables such as MAXJOBS and similar command line options.

    +

    When using the special value 0, BuildStream will allocate the number of threads +available on the host and limit this with a hard coded value of 8, which was +found to be an optimial number when building even on hosts with many cores.

    +
    • +

  • retry-failed

    +

    Try to build elements for which a failed build artifact is found when running +bst build.

    +

    This is useful in the case that a build has failed due to insufficient resources +such as memory or disk space.

    +
    • +

  • dependencies

    +

    This instructs what dependencies of the target elements should be built, valid +values for this attribute are:

    +
      +

    • none: Only build elements required to generate the expected target artifacts

      • +

    • all: Build elements even if they are build dependencies of artifacts which are already cached

      • +
    +
    • +
+
+
+
+

Fetch controls

+

Some aspects about how sources get fetched can be controlled by attributes of the fetch +dictionary at the toplevel, like so:

+
#
+# Fetch controls
+#
+fetch:
+
+  #
+  # Don't allow fetching from project defined alias or mirror URIs
+  #
+  source: user
+
+
+
+

Attributes

+ +
+
+
+

Track controls

+

Some aspects about how sources get tracked can be controlled by attributes of the track +dictionary at the toplevel, like so:

+
#
+# Track controls
+#
+track:
+
+  #
+  # Only track sources for new refs from project defined default alias URIs
+  #
+  source: aliases
+
+
+
+

Attributes

+ +
+
+
+

Logging controls

+

Various aspects of how BuildStream presents output and UI can be controlled with +attributes of the toplevel logging dictionary, like so:

+
#
+# Control logging output
+#
+logging:
+
+  #
+  # Whether to be verbose
+  #
+  verbose: True
+
+
+
+

Attributes

+
    +

  • verbose

    +

    Whether to use verbose logging.

    +
    • +

  • debug

    +

    Whether to print messages related to debugging BuildStream itself.

    +
    • +

  • key-length

    +

    When displaying abbreviated cache keys, this controls the number of characters +of the cache key which should be printed.

    +
    • +

  • throttle-ui-updates

    +

    Whether the throttle updates to the status bar in interactive mode. If set to True, +then the status bar will be updated once per second.

    +
    • +

  • error-lines

    +

    The maximum number of lines to print in the main logging output related to an +error processing an element, these will be the last lines found in the relevant +element’s stdout and stderr.

    +
    • +

  • message-lines

    +

    The maximum number of lines to print in a detailed message sent to the main logging output.

    +
    • +

  • element-format

    +

    The default format to use when printing out elements in bst show +output, and when printing the pipeline summary at the beginning of sessions.

    +

    The format is specified as a string containing variables which will be expanded +in the resulting string, variables must be specified using a leading percent sign +and enclosed in curly braces, a colon can be specified in the variable to perform +python style string alignments, e.g.:

    +
    logging:
+
+  #
+  # Set the element format
+  #
+  element-format: |
+
+    %{state: >12} %{full-key} %{name} %{workspace-dirs}
+
    +
    +

    Variable names which can be used in the element format consist of:

    +
      +

    • name

      +

      The element path, which is the name of the element including +any leading junctions.

      +
      • +

    • key

      +

      The abbreviated cache key, the length of which is controlled by the key-length logging configuration.

      +
      • +

    • full-key

      +

      The full cache key.

      +
      • +

    • state

      +

      The element state, this will be formatted as one of the following:

      +
        +

      • no reference: If the element still needs to be tracked.

        • +

      • junction: If the element is a junction and as such does not have any relevant state.

        • +

      • failed: If the element has been built and the build has failed.

        • +

      • cached: If the element has been successfully built and is present in the local cache.

        • +

      • fetch needed: If the element cannot be built yet because the sources need to be fetched.

        • +

      • buildable: If the element has all of its sources and build dependency artifacts cached locally.

        • +

      • waiting: If the element has all of its sources cached but its build dependencies are not yet locally cached.

        • +
      +
      • +

    • config

      +

      The element configuration, formatted in YAML.

      +
      • +

    • vars

      +

      The resolved element variables, formatted as a simple YAML dictionary.

      +
      • +

    • env

      +

      The resolved environment variables, formatted as a simple YAML dictionary.

      +
      • +

    • public

      +

      The resolved public data, formatted in YAML.

      +
      • +

    • workspaced

      +

      If the element has an open workspace, this will expand to the string “(workspaced)”, otherwise +it will expand to an empty string.

      +
      • +

    • workspace-dirs

      +

      If the element has an open workspace, this will expand to the workspace directory, prefixed with +the text “Workspace: “, otherwise it will expand to an empty string.

      +
      • +

    • deps

      +

      A list of the element paths of all dependency elements.

      +
      • +

    • build-deps

      +

      A list of the element paths of all build dependency elements.

      +
      • +

    • runtime-deps

      +

      A list of the element paths of all runtime dependency elements.

      +
      • +
    +
    • +

  • message-format

    +

    The format to use for messages being logged in the aggregated main logging output.

    +

    Similarly to the element-format, The format is specified as a string containing variables which +will be expanded in the resulting string, and variables must be specified using a leading percent sign +and enclosed in curly braces, e.g.:

    +
    logging:
+
+  #
+  # Set the message format
+  #
+  message-format: |
+
+    [%{elapsed}][%{key}][%{element}] %{action} %{message}
+
    +
    +

    Variable names which can be used in the element format consist of:

    +
      +

    • elapsed

      +

      If this message announces the completion of (successful or otherwise) of an activity, then +this will expand to a time code showing how much time elapsed for the given activity, in +the format: HH:MM:SS, otherwise an empty time code will be displayed in the format: +--:--:--.

      +
      • +

    • elapsed-us

      +

      Similar to the elapsed variable, however the timecode will include microsecond precision.

      +
      • +

    • wallclock

      +

      This will display a timecode for each message displaying the local wallclock time, in the +format HH:MM:SS.

      +
      • +

    • wallclock-us

      +

      Similar to the wallclock variable, however the timecode will include microsecond precision.

      +
      • +

    • key

      +

      The abbreviated cache key of the element the message is related to, the length of which is controlled +by the key-length logging configuration.

      +

      If the message in question is not related to any element, then this will expand to whitespace +of equal length.

      +
      • +

    • element

      +

      This will be formatted to an indicator consisting of the type of activity which is being +performed on the element (e.g. “build” or “fetch” etc), and the element path +of the element this message is related to.

      +

      If the message in question is not related to any element, then a string will be formatted +to indicate that this message is related to a core activity instead.

      +
      • +

    • action

      +

      A classifier of the type of message this is, the possible values this will expand to are:

      +
        +

      • DEBUG

        +

        This is a message related to debugging BuildStream itself

        +
        • +

      • STATUS

        +

        A message showing some detail of what is currently happening, this message will not +be displayed unless verbose output is enabled.

        +
        • +

      • INFO

        +

        An informative message, this may be printed for instance when discovering a new +ref for source code when running bst source track.

        +
        • +

      • WARN

        +

        A warning message.

        +
        • +

      • ERROR

        +

        An error message.

        +
        • +

      • BUG

        +

        A bug happened in BuildStream, this will usually be accompanied by a python stack trace.

        +
        • +

      • START

        +

        An activity related to an element started.

        +

        Any START message will always be accompanied by a later SUCCESS, FAILURE +or SKIPPED message.

        +
        • +

      • SUCCESS

        +

        An activity related to an element completed successfully.

        +
        • +

      • FAILURE

        +

        An activity related to an element failed.

        +
        • +

      • SKIPPED

        +

        After strating this activity, it was discovered that no work was needed and +the activity was skipped.

        +
        • +
      +
      • +

    • message

      +

      The brief message, or the path to the corresponding log file, will be printed here.

      +

      When this is a scheduler related message about the commencement or completion of +an element related activity, then the path to the corresponding log for that activity +will be printed here.

      +

      If it is a message issued for any other reason, then the message text will be formatted here.

      +
      • +
    +
    +

    Note

    +

    Messages issued by the core or by plugins are allowed to provide detailed accounts, these +are the indented multiline messages which sometimes get displayed in the main aggregated +logging output, and will be printed regardless of the logging message-format value.

    +
    +
    • +
+
+
+
+

Remote services

+

BuildStream can be configured to cooperate with remote caches and +execution services.

+
+

Authentication

+

BuildStream supports end to end encryption when communicating with remote +services.

+

All remote service configuration blocks come with an optional auth +configuration block which allows one to specify the certificates +and keys required for encrypted traffic.

+

The auth configuration block looks like this:

+
auth:
+  server-cert: server.crt
+  client-cert: client.crt
+  client-key: client.key
+  access-token: access.token
+  access-token-reload-interval: 60
+
+
+
+

Attributes

+
    +

  • server-cert

    +

    The server certificate is used to verify the identity of the server instead +of using the CA store from the operating system for verification.

    +
    • +

  • client-cert

    +

    The client certificate is used by the remote server to authenticate the client.

    +
    • +

  • client-key

    +

    The private client key corresponding to the specified client certificate.

    +
    • +

  • access-token

    +

    The path to a token for optional HTTP bearer authentication.

    +
    • +

  • access-token-reload-interval

    +

    The reload interval in minutes for the specified access token. +If not specified, automatic reloading is disabled.

    +
    • +
+

Remote cache services may allow downloading artifacts and sources without +authentication, in which case only server-cert is required for secure access +(or no attributes at all if the CA store from the OS can be used).

+

However, remote cache services should normally not allow uploading artifacts +or sources without authentication. Authentication can be configured by setting +access-token or both client-key and client-cert.

+
+
+
+

Cache servers

+

BuildStream supports two types of cache servers, source cache servers +and artifact cache servers. These services allow you +to store sources and build artifacts for later reuse, and share them among your +peers.

+
+

Important

+

Storing and indexing

+

Cache servers are split into two separate services, the index and the storage. +Sometimes these services are provided by the same server, and sometimes it is desirable +to use different cache servers for indexing and storing data.

+

In simple setups, it is possible to use the same cache server for indexing and storing +of both sources and artifacts. However, when using remote execution +it is recommended to use the remote execution build cluster’s storage-service as the storage +service of your cache servers, which may require setting up your index service separately.

+

When configuring cache servers, BuildStream will require both storage and indexing capabilities, +otherwise no attempt will be made to fetch or push data to and from cache servers.

+
+

Cache server configuration is declared in the following way:

+
override-project-caches: false
+servers:
+- url: https://cache-server.com/cache:11001
+  instance-name: main
+  type: all
+  push: true
+  auth:
+    server-cert: server.crt
+    client-cert: client.crt
+    client-key: client.key
+
+
+
+

Attributes

+
    +

  • override-project-caches

    +

    Whether this user configuration overrides the project recommendations for +artifact caches or source caches.

    +

    If this is false (which is the default), then project recommended cache +servers will be observed after user specified caches.

    +
    • +

  • servers

    +

    This is the list of cache servers in the configuration block, every entry +in the block represents a server which will be accessed in the specified order.

    +
      +

    • url

      +

      Indicates the http or https url and optionally the port number of +where the cache server is located.

      +
      • +

    • instance-name

      +

      Instance names separate different shards on the same endpoint (url).

      +

      The instance name is optional, and not all cache server implementations support +instance names. The instance name should be given to you by the +service provider of each service.

      +
      • +

    • type

      +

      The type of service you intend to use this cache server for. If unspecified, +the default value for this field is all.

      +
        +

      • storage

        +

        Use this cache service for storage.

        +
        • +

      • index

        +

        Use this cache service for index content expected to be present in one +or more storage services.

        +
        • +

      • all

        +

        Use this cache service for both indexing and storing data.

        +
        • +
      +
      • +

    • push

      +

      Set this to true if you intend to upload data to this cache server.

      +

      Normally this requires additional credentials in the auth field.

      +
      • +

    • auth

      +

      The authentication attributes to connect to +this server.

      +
      • +
    +
    • +
+
+
+

Cache server lists

+

Cache servers are always specified as lists in the configuration, this allows +index and storage services to be declared separately, and also allows for +some redundancy.

+

Example:

+
- url: https://cache-server-1.com/index
+  type: index
+- url: https://cache-server-1.com/storage
+  type: storage
+- url: https://cache-server-2.com
+  type: all
+
+
+

When downloading data from a cache server, BuildStream will iterate over each +index service one by one until it finds the reference to the data it is looking +for, and then it will iterate over each storage service one by one, downloading +the referenced data until all data is downloaded.

+

When uploading data to a cache server, BuildStream will first upload the data to +each storage service which was configured with the push attribute, and +upon successful upload, it will proceed to upload the references to the uploaded +data to each index service in the list.

+
+
+
+

Artifact cache servers

+

Using artifact cache servers is an essential means of +build avoidance, as it will allow you to avoid building an element which has already +been built and uploaded to a common artifact server.

+

Artifact cache servers can be declared in different ways, with differing priorities.

+
+

Command line

+

Various commands which involve connecting to artifact servers allow +specifying remotes, remotes specified +on the command line replace all user configuration.

+
+
+

Global caches

+

To declare the global artifact server list, use the artifacts key at the +toplevel of the user configuration.

+
#
+# Configure a global artifact server for pushing and pulling artifacts
+#
+artifacts:
+  override-project-caches: false
+  servers:
+  - url: https://artifacts.com/artifacts:11001
+    push: true
+    auth:
+      server-cert: server.crt
+      client-cert: client.crt
+      client-key: client.key
+
+
+
+
+

Project overrides

+

To declare artifact servers lists for individual projects, declare them +in the project specific section of +the user configuration.

+

Artifact server lists declared in this section will only be used for +elements belonging to the specified project, and will be used instead of +artifact cache servers declared in the global caches.

+
#
+# Configure an artifact server for pushing and pulling artifacts from project "foo"
+#
+projects:
+  foo:
+    artifacts:
+      override-project-caches: false
+      servers:
+      - url: https://artifacts.com/artifacts:11001
+        push: true
+        auth:
+          server-cert: server.crt
+          client-cert: client.crt
+          client-key: client.key
+
+
+
+
+

Project recommendations

+

Projects can recommend artifact cache servers in their +individual project configuration files.

+

These will only be used for elements belonging to their respective projects, and +are the lowest priority configuration.

+
+
+
+

Source cache servers

+

Using source cache servers enables BuildStream to cache +source code referred to by your project and share those sources with peers who have +access to the same source cache server.

+

This can optimize your build times in the case that it is determined that an element needs +to be rebuilt because of changes in the dependency graph, as BuildStream will first attempt +to download the source code from the cache server before attempting to obtain it from an +external source, which may suffer higher latencies.

+

Source cache servers can be declared in different ways, with differing priorities.

+
+

Command line

+

Various commands which involve connecting to source cache servers allow +specifying remotes, remotes specified +on the command line replace all user configuration.

+
+
+

Global caches

+

To declare the global source cache server list, use the source-caches key at the +toplevel of the user configuration.

+
#
+# Configure a global source cache server for pushing and pulling sources
+#
+source-caches:
+  override-project-caches: false
+  servers:
+  - url: https://sources.com/sources:11001
+    push: true
+    auth:
+      server-cert: server.crt
+      client-cert: client.crt
+      client-key: client.key
+
+
+
+
+

Project overrides

+

To declare source cache servers lists for individual projects, declare them +in the project specific section of +the user configuration.

+

Source cache server lists declared in this section will only be used for +elements belonging to the specified project, and will be used instead of +source cache servers declared in the global caches.

+
#
+# Configure a source cache server for pushing and pulling sources from project "foo"
+#
+projects:
+  foo:
+    source-caches:
+      override-project-caches: false
+      servers:
+      - url: https://sources.com/sources:11001
+        push: true
+        auth:
+          server-cert: server.crt
+          client-cert: client.crt
+          client-key: client.key
+
+
+
+
+

Project recommendations

+

Projects can recommend source cache servers in their +individual project configuration files.

+

These will only be used for elements belonging to their respective projects, and +are the lowest priority configuration.

+
+
+
+

Remote execution

+

BuildStream supports building remotely using the +Google Remote Execution API (REAPI)..

+

You can configure the remote execution services globally in your user configuration +using the remote-execution key, like so:

+
remote-execution:
+  execution-service:
+    url: http://execution.fallback.example.com:50051
+    instance-name: main
+  storage-service:
+    url: https://storage.fallback.example.com:11002
+    instance-name: main
+    auth:
+      server-cert: /keys/server.crt
+      client-cert: /keys/client.crt
+      client-key: /keys/client.key
+  action-cache-service:
+    url: http://cache.flalback.example.com:50052
+    instance-name: main
+
+
+
+

Attributes

+
    +

  • execution-service

    +

    A service configuration specifying +how to connect with the main execution service, this service is the main controlling +entity in a remote execution build cluster.

    +
    • +

  • storage-service

    +

    A service configuration specifying +how to connect with the Content Addressable Storage service, this is where build +input and output is stored on the remote execution build cluster.

    +

    This service is compatible with the storage service offered by +cache servers.

    +

    This is optional if a storage-service is configured in the +cache configuration, in which case actual file +contents of build outputs will only be downloaded as needed, e.g. on +bst artifact checkout.

    +
    • +

  • action-cache-service

    +

    A service configuration specifying +how to connect with the action cache, this service stores information about +activities which clients request be performed by workers on the remote execution +build cluster, and results of completed operations.

    +

    This service is optional in a remote execution build cluster, if your remote +execution service provides an action cache, then you should configure it here.

    +
    • +
+
+
+

Remote execution service configuration

+

Each of the distinct services are described by the same configuration block, +which looks like this:

+
url: https://storage.fallback.example.com:11002
+instance-name: main
+auth:
+  server-cert: /keys/server.crt
+  client-cert: /keys/client.crt
+  client-key: /keys/client.key
+
+
+

Attributes:

+
    +

  • url

    +

    Indicates the http or https url and optionally the port number of +where the service is located.

    +
    • +

  • instance-name

    +

    The instance name is optional. Instance names separate different shards on +the same endpoint (url). The instance name should be given to you by the +service provider of each service.

    +

    Not all service providers support instance names.

    +
    • +

  • auth

    +

    The authentication attributes to connect to +this server.

    +
    • +
+
+
+
+
+

Project specific values

+

The projects key can be used to specify project specific configurations, +the supported configurations on a project wide basis are listed here.

+
+

Strict build plan

+

The strict build plan option decides whether you want elements +to rebuild when their dependencies have changed. This is enabled +by default, but recommended to turn off in developer scenarios where +you might want to build a large system and test it quickly after +modifying some low level component.

+

Example

+
projects:
+  project-name:
+    strict: False
+
+
+
+

Note

+

It is always possible to override this at invocation time using +the --strict and --no-strict command line options.

+
+
+
+

Mirrors

+

Project defined mirrors, can be overridden +with user configuration. This is helpful when you need to mirror all of the source +code used by subprojects and ensure that your project can be built in perpetuity.

+

Example

+
projects:
+  project-name:
+    mirrors:
+    - name: middle-earth
+      aliases:
+        foo:
+        - http://www.middle-earth.com/foo/1
+        - http://www.middle-earth.com/foo/2
+        bar:
+        - http://www.middle-earth.com/bar/1
+        - http://www.middle-earth.com/bar/2
+    - name: oz
+      aliases:
+        foo:
+        - http://www.oz.com/foo
+        bar:
+        - http://www.oz.com/bar
+
+
+
+
+

Default mirror

+

When using mirrors, one can specify which +mirror should be used first.

+

Example

+
projects:
+  project-name:
+    default-mirror: oz
+
+
+
+

Note

+

It is possible to override this at invocation time using the +--default-mirror command-line option.

+
+
+
+

Project options

+

One can specify values to use for project options for the projects +you use here, this avoids needing to specify the options on the command line every time.

+

Example

+
projects:
+
+  #
+  # Configure the debug flag offered by `project-name`
+  #
+  project-name:
+    options:
+      debug-build: True
+
+
+
+
+

Source cache servers

+

As already described in the section concerning configuration of +source cache servers, these can be specified on a per project basis.

+
+
+

Artifact cache servers

+

As already described in the section concerning configuration of +artifact cache servers, these can be specified on a per project basis.

+
+
+

Remote execution configuration

+

Following the same format as the toplevel remote execution configuration, +the global configuration can be overridden on a per project basis in this project override section.

+

Example

+
projects:
+
+  project-name:
+
+    #
+    # If `project-name` is built as the toplevel project in this BuildStream session,
+    # then use this remote execution configuration instead of any globally defined
+    # remote execution configuration.
+    #
+    remote-execution:
+      execution-service:
+        url: http://execution.example.com:50051
+        instance-name: main
+
+
+
+

Note

+

Only one remote execution service will be considered for any invocation of BuildStream.

+

If you are building a project which has a junction into another subproject for which you have +specified a project specific remote execution service for in your user configuration, then +it will be ignored in the context of building that toplevel project.

+
+
+
+
+

Default configuration

+

The default BuildStream configuration is specified here for reference:

+
+
#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# Default BuildStream user configuration.
+
+#
+#    Work Directories
+#
+
+# Location to store sources
+sourcedir: ${XDG_CACHE_HOME}/buildstream/sources
+
+# Root location for other directories in the cache
+cachedir: ${XDG_CACHE_HOME}/buildstream
+
+# Location to store build logs
+logdir: ${XDG_CACHE_HOME}/buildstream/logs
+
+# Default root location for workspaces, blank for no default set.
+workspacedir: .
+
+#
+#    Cache
+#
+cache:
+
+  # Use as much space as possible
+  quota: infinity
+
+  # Keep 5% of disk space available
+  reserved-disk-space: 5%
+
+  # Retain 80% of the cache on cleanup
+  low-watermark: 80%
+
+  # Whether to pull build trees when downloading element artifacts
+  pull-buildtrees: False
+
+  # Whether to cache build trees on artifact creation
+  #
+  cache-buildtrees: auto
+
+
+#
+#    Scheduler
+#
+scheduler:
+
+  # Maximum number of simultaneous downloading tasks.
+  fetchers: 10
+
+  # Maximum number of simultaneous build tasks.
+  builders: 4
+
+  # Maximum number of simultaneous uploading tasks.
+  pushers: 4
+
+  # Maximum number of retries for network tasks.
+  network-retries: 2
+
+  # Control what to do when a task fails, if not running in
+  # interactive mode
+  #
+  on-error: quit
+
+
+#
+# Build related configuration
+#
+build:
+
+  #
+  # Maximum number of jobs to run per build task.
+  #
+  max-jobs: 0
+
+  #
+  # Try to build elements for which a failed build artifact is found
+  #
+  retry-failed: False
+
+  #
+  # Control which dependencies to build
+  #
+  dependencies: none
+
+
+#
+# Source fetch related configuration
+#
+fetch:
+
+  #
+  # Which URIs are allowed to be accessed
+  #
+  source: all
+
+
+#
+# Source track related configuration
+#
+track:
+
+  #
+  # Which URIs are allowed to be accessed
+  #
+  source: aliases
+
+
+#
+#    Logging
+#
+logging:
+
+  # The abbreviated cache key length to display in the UI
+  key-length: 8
+
+  # Whether to show extra detailed messages
+  verbose: True
+
+  # Maximum number of lines to print from the
+  # end of a failing build log
+  error-lines: 20
+
+  # Maximum number of lines to print in a detailed
+  # message on the console or in the master log (the full
+  # messages are always recorded in the individual build
+  # logs)
+  message-lines: 20
+
+  # Whether to enable debugging messages
+  debug: False
+
+  # Format string for printing the pipeline at startup, this
+  # also determines the default display format for `bst show`
+  #
+  element-format: |
+
+    %{state: >12} %{full-key} %{name} %{workspace-dirs}
+
+  # Format string for log messages.
+  #
+  message-format: |
+
+    [%{elapsed}][%{key}][%{element}] %{action} %{message}
+
+  # Limit bst console output update rate to 1Hz where applicable
+  throttle-ui-updates: True
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_configuring_cache_server.html b/using_configuring_cache_server.html new file mode 100644 index 000000000..4014e8e13 --- /dev/null +++ b/using_configuring_cache_server.html @@ -0,0 +1,229 @@ + + + + + + + + + Configuring Cache Servers — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Configuring Cache Servers

+

BuildStream caches the results of builds in a local artifact cache, and will +avoid building an element if there is a suitable build already present in the +local artifact cache. Similarly it will cache sources and avoid pulling them if +present in the local cache. See caches for more details.

+

In addition to the local caches, you can configure one or more remote caches and +BuildStream will then try to pull a suitable object from one of the remotes, +falling back to performing a local build or fetching a source if needed.

+

On the client side, cache servers are declared and configured in +user configuration, and since it is typical +for projects to maintain their own cache servers, it is also possible for +projects to provide recommended artifact cache servers +and source cache servers through project +configuration, so that downstream users can download from services +provided by upstream projects by default.

+
+

Setting up a remote cache

+

BuildStream relies on the ContentAddressableStorage protocol +in order to exchange data with remote services, in concert with the remote asset protocol +in order to assign symbolic labels (such as artifact names) to identify +stored content. As such, BuildStream is able to function with any implementations of these +two services.

+
+
+

Known implementations

+

Here are some details about known open source implementations of the required protocols

+
+

Buildbarn

+

The Buildbarn project provides a remote execution +service implementation for use in build tooling such as BuildStream, Bazel +and recc, the bb-storage +and bb-remote-asset services are tested +to work as cache service for BuildStream’s artifact and source caches.

+

A simple configuration to spin up the service using docker compose follows:

+
##
+# Buildbarn Compose manifest for BuildStream.
+#
+# Spins-up a unnamed and unauthenticated cache server:
+#  - STORAGE at http://localhost:7982
+#  - INDEX at: http://localhost:7981
+#
+# BuildStream configuration snippet:
+#
+# artifacts:
+#   - url: https://localhost:7981
+#     type: index
+#     push: true
+#   - url: https://localhost:7982
+#     type: storage
+#     push: true
+#
+# Basic usage:
+#  - docker-compose -f ci.buildbarn.yml up
+#  - docker-compose -f ci.buildbarn.yml down
+
+version: '3.4'
+
+services:
+  bb-asset:
+    image: buildbarn/bb-remote-asset:20200903T103837Z-90136c4
+    command: /config/asset.jsonnet
+    restart: unless-stopped
+    ports:
+    - "7981:7981"
+    volumes:
+    - type: volume
+      source: assets
+      target: /storage
+    - type: bind
+      source: ./buildbarn-config/
+      target: /config
+
+  bb-storage:
+    image: buildbarn/bb-storage:20200816T115912Z-d83e1f0
+    command: /config/storage.jsonnet
+    restart: unless-stopped
+    ports:
+    - "7982:7982"
+    volumes:
+    - type: volume
+      source: cas
+      target: /cas
+    - type: bind
+      source: ./buildbarn-config/
+      target: /config
+
+volumes:
+  assets:
+  cas:
+
+
+

Visit the bb-storage and +bb-remote-asset project pages to +find more documentation about setting up services with authentication enabled.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_configuring_remote_execution.html b/using_configuring_remote_execution.html new file mode 100644 index 000000000..92d2a4759 --- /dev/null +++ b/using_configuring_remote_execution.html @@ -0,0 +1,251 @@ + + + + + + + + + Remote Execution Servers — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Remote Execution Servers

+

BuildStream supports building remotely using the +Google Remote Execution API (REAPI)., which +has various known implementations.

+

Some of these implementations include:

+ +

These various implementations implement the Google Remote Execution API (REAPI) to various degrees as these projects +have different priorities.

+

On the client side, the remote execution service to use can be +specified in the user configuration.

+
+

BuildStream specific requirements

+

In order for BuildStream to work correctly with a remote execution cluster, there +are a couple of requirements that implementation needs to meet.

+
+

Implementation of platform properties

+

The remote execution service must properly implement platform properties.

+

This is crucial because BuildStream needs to be guaranteed the correct operating +system and ISA which it requests from the service.

+
+
+

Staging the input root as the filesystem root

+

BuildStream requires that the input root given to the remote execution service +be treated as the absolute filesystem root.

+

This is because BuildStream provides guarantees that all build dependencies, including +the base runtime and compilers, are defined by elements and run within a sandboxed and +isolated build environment, but the REAPI +was originally developped without this determinism and control in mind. Instead, typically +it is up to the user to configure a cluster to use a docker image to build payloads +with, rather than allowing the REAPI client to control the entire sandbox.

+

Unfortunately the ability to dictate that the input root be treated as the filesystem root +in a container on remote workers in the cluster is not yet standardized in the REAPI protocol.

+
+

Note

+

The input root is referred to as the input_root_digest member of the Action message +as defined in the protocol

+
+
+
+
+

Example working configuration

+

A simple configuration to spin up the BuildGrid service using +docker compose follows:

+
##
+# BuildGrid Compose manifest for BuildStream.
+#
+# Spins-up a unnamed and unauthenticated grid:
+#  - Controller + CAS + AC at http://localhost:50051
+#  - Ref. + CAS at: http://localhost:50052
+#
+# BuildStream configuration snippet:
+#
+#    artifacts:
+#      url: http://localhost:50052
+#      push: true
+#    remote-execution:
+#      execution-service:
+#        url: http://localhost:50051
+#      action-cache-service:
+#        url: http://localhost:50051
+#      storage-service:
+#        url: http://localhost:50051
+#
+# Basic usage:
+#  - docker-compose -f ci.buildgrid.yml up
+#  - docker-compose -f ci.buildgrid.yml down
+#
+version: "3.2"
+
+services:
+  controller:
+    image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
+    command: [
+      "bgd", "server", "start", "-v",
+      "/etc/buildgrid/default.yml"]
+    ports:
+      - 50051:50051
+    networks:
+      - grid
+
+  bot:
+    image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildbox:nightly
+    command: [
+      "sh", "-c", "sleep 15 && ( buildbox-casd --cas-remote=http://controller:50051 /var/lib/buildgrid/cache & buildbox-worker --request-timeout=30 --bots-remote=http://controller:50051 --cas-remote=unix:/var/lib/buildgrid/cache/casd.sock --buildbox-run=buildbox-run-bubblewrap --platform OSFamily=linux --platform ISA=x86-64 --verbose )"]
+    privileged: true
+    volumes:
+      - type: volume
+        source: cache
+        target: /var/lib/buildgrid/cache
+    depends_on:
+      - controller
+    networks:
+      - grid
+
+  storage:
+    image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
+    command: [
+      "bgd", "server", "start", "-v",
+      "/etc/buildgrid/artifacts.yml"]
+    ports:
+      - 50052:50052
+    networks:
+      - grid
+
+networks:
+  grid:
+    driver: bridge
+
+volumes:
+  cache:
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_developing.html b/using_developing.html new file mode 100644 index 000000000..a1e289755 --- /dev/null +++ b/using_developing.html @@ -0,0 +1,140 @@ + + + + + + + + + Developing — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Developing

+

This section covers the features BuildStream provides for users +to develop their integrated products, as well as developping their +modules in the context of an integrated product.

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_examples.html b/using_examples.html new file mode 100644 index 000000000..300275fec --- /dev/null +++ b/using_examples.html @@ -0,0 +1,140 @@ + + + + + + + + + Examples — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Examples

+

This page contains documentation for real examples of BuildStream projects, +described step by step. All run under CI, so you can trust they are +maintained and work as expected.

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_handling_files.html b/using_handling_files.html new file mode 100644 index 000000000..7993b7121 --- /dev/null +++ b/using_handling_files.html @@ -0,0 +1,141 @@ + + + + + + + + + Handling files — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Handling files

+

This section covers the various aspects related to finer grained +handling of files.

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_junctions.html b/using_junctions.html new file mode 100644 index 000000000..6a6676dc6 --- /dev/null +++ b/using_junctions.html @@ -0,0 +1,141 @@ + + + + + + + + + Combining projects — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Combining projects

+

This section covers the features which allow BuildStream projects to +depend on eachother.

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/using_tutorial.html b/using_tutorial.html new file mode 100644 index 000000000..6f4851fba --- /dev/null +++ b/using_tutorial.html @@ -0,0 +1,146 @@ + + + + + + + + + Getting started — BuildStream 2.3.0+26.g65e2074ad documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Getting started

+

This is a step by step walkthrough meant help the user quickly get +familiar with the fundamentals of creating and using BuildStream +projects.

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file