This document describes what Bottlerocket variants are and how they are built.
In the Background section, we discuss the motivation for variants.
In the Variants section, we list the variants that exist today.
In the Development section, we provide a short guide for adding a new variant.
Bottlerocket is purpose-built for hosting containers. It can run one of several container orchestrator agents. It is also image-based and does not include a package manager for customization at runtime.
Conceptually, each image could include all orchestrator agents, but that would conflict with our design goals. We want to keep the footprint of Bottlerocket as small as possible for security and performance reasons. Instead, we make different variants available for use, each with its own set of software and API settings.
A variant is essentially a list of packages to install, plus a model that defines the API. The documentation for packages covers how to create a package. Information about API settings for variants can be found in the models documentation.
Bottlerocket variants ingest TOML-formatted user data from various sources in a predefined order.
All variants first attempt to read user data from /var/lib/bottlerocket/user-data.toml.
AWS variants then retrieve user data from IMDS.
VMware variants will attempt to read user data from a mounted CD-ROM (from a file named "user-data" or from an OVF file), and then from VMware's guestinfo interface.
If a setting is defined in more than one source, the value in later sources will override earlier values. For example, in a VMware variant, settings read from the guestinfo interface will override settings from CD-ROM, and settings from CD-ROM will override settings from the file.
See Update Policy in the Security Features document for information on when and how Bottlerocket applies security patches to variants.
The aws-k8s-1.23 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.23, 1.24, and 1.25 clusters.
The aws-k8s-1.23-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.23, 1.24, and 1.25 clusters.
The aws-k8s-1.24 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.24, 1.25, and 1.26 clusters.
The aws-k8s-1.24-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.24, 1.25, and 1.26 clusters.
The aws-k8s-1.25 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.25, 1.26, 1.27, and 1.28 clusters.
The aws-k8s-1.25-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.25, 1.26, 1.27, and 1.28 clusters.
The aws-k8s-1.26 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.26, 1.27, 1.28, and 1.29 clusters.
The aws-k8s-1.26-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.26, 1.27, 1.28, and 1.29 clusters.
The aws-k8s-1.27 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.27, 1.28, 1.29, and 1.30 clusters.
The aws-k8s-1.27-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.27, 1.28, 1.29, and 1.30 clusters.
The aws-k8s-1.28 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.28, 1.29, 1.30, and 1.31 clusters.
The aws-k8s-1.28-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.28, 1.29, 1.30, and 1.31 clusters.
The aws-k8s-1.29 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.29, 1.30, 1.31, and 1.32 clusters.
The aws-k8s-1.29-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.29, 1.30, 1.31 and 1.32 clusters.
The aws-k8s-1.30 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.30, 1.31, 1.32, and 1.33 clusters.
The aws-k8s-1.30-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.30, 1.31, 1.32, and 1.33 clusters.
The aws-k8s-1.31 variant includes the packages needed to run a Kubernetes node in AWS. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.31, 1.32, 1.33, and 1.34 clusters.
The aws-k8s-1.31-nvidia variant includes the packages needed to run a Kubernetes node in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs. It supports self-hosted clusters and clusters managed by EKS.
This variant is compatible with Kubernetes 1.31, 1.32, 1.33, and 1.34 clusters.
The aws-ecs-1 variant includes the packages needed to run an Amazon ECS container instance in AWS.
The aws-ecs-1-nvidia variant includes the packages needed to run an Amazon ECS container instance in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs.
The aws-ecs-2 variant includes the packages needed to run an Amazon ECS container instance in AWS.
The aws-ecs-2-nvidia variant includes the packages needed to run an Amazon ECS container instance in AWS. It also includes the required packages to configure containers to leverage NVIDIA GPUs.
The aws-dev variant has useful packages for local development of the OS. It includes tools for troubleshooting as well as Docker for running containers. User data will be read from IMDS.
The vmware-dev variant has useful packages for local development of the OS, and is intended to run as a VMware guest. It includes tools for troubleshooting as well as Docker for running containers.
The vmware-k8s-1.28 variant includes the packages needed to run a Kubernetes worker node as a VMware guest. It supports self-hosted clusters.
This variant is compatible with Kubernetes 1.28, 1.29, 1.30 and 1.31 clusters.
The vmware-k8s-1.29 variant includes the packages needed to run a Kubernetes worker node as a VMware guest. It supports self-hosted clusters.
This variant is compatible with Kubernetes 1.29, 1.30, 1.31, and 1.32 clusters.
The vmware-k8s-1.30 variant includes the packages needed to run a Kubernetes worker node as a VMware guest. It supports self-hosted clusters.
This variant is compatible with Kubernetes 1.30, 1.31, 1.32, and 1.33 clusters.
The vmware-k8s-1.31 variant includes the packages needed to run a Kubernetes worker node as a VMware guest. It supports self-hosted clusters.
This variant is compatible with Kubernetes 1.31, 1.32, 1.33, and 1.34 clusters.
The metal-dev variant has useful packages for local development of the OS and is intended to run bare metal. It includes tools for troubleshooting as well as Docker for running containers.
The metal-k8s-1.28 variant includes the packages needed to run a Kubernetes node on bare metal. It supports self-hosted clusters.
This variant is compatible with Kubernetes 1.28, 1.29, 1.30, and 1.31 clusters.
The metal-k8s-1.29 variant includes the packages needed to run a Kubernetes node on bare metal. It supports self-hosted clusters.
This variant is compatible with Kubernetes 1.29, 1.30, 1.31, and 1.32 clusters.
The aws-k8s-1.15 variant included the packages needed to run a Kubernetes node in AWS. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.15, 1.16, and 1.17 clusters. It reached end-of-life on May 3, 2021.
Upstream support for Kubernetes 1.15 has ended and this variant will no longer be supported in Bottlerocket releases.
The aws-k8s-1.16 variant included the packages needed to run a Kubernetes node in AWS. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.16, 1.17, and 1.18 clusters. It reached end-of-life on July 25, 2021.
Upstream support for Kubernetes 1.16 has ended and this variant will no longer be supported in Bottlerocket releases.
The aws-k8s-1.17 variant included the packages needed to run a Kubernetes node in AWS. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.17, 1.18, and 1.19 clusters. It reached end-of-life on November 2, 2021.
Upstream support for Kubernetes 1.17 has ended and this variant will no longer be supported in Bottlerocket releases.
The aws-k8s-1.18 variant included the packages needed to run a Kubernetes node in AWS. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.18, 1.19, and 1.20 clusters. It reached end-of-life on March 31st, 2022.
Upstream support for Kubernetes 1.18 has ended and this variant will no longer be supported in Bottlerocket releases.
The aws-k8s-1.19 variant included the packages needed to run a Kubernetes node in AWS. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.19, 1.20, and 1.21 clusters. It reached end-of-life on August 1st, 2022.
Upstream support for Kubernetes 1.19 has ended and this variant will no longer be supported in Bottlerocket releases.
The aws-k8s-1.20 variant included the packages needed to run a Kubernetes node in AWS. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.20, 1.21, and 1.22 clusters. It reached end-of-life on November 1st, 2022.
Upstream support for Kubernetes 1.20 has ended and this variant will no longer be supported in Bottlerocket releases.
The vmware-k8s-1.20 variant included the packages needed to run a Kubernetes worker node as a VMware guest. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.20, 1.21, and 1.22 clusters.
The aws-k8s-1.21 variant included the packages needed to run a Kubernetes node in AWS. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.21, 1.22, and 1.23 clusters.
The aws-k8s-1.21-nvidia variant included the packages needed to run a Kubernetes node in AWS. It also included the required packages to configure containers to leverage NVIDIA GPUs. It supported self-hosted clusters and clusters managed by EKS. This variant was compatible with Kubernetes 1.21, 1.22, and 1.23 clusters.
The metal-k8s-1.21 variant included the packages needed to run a Kubernetes node on bare metal. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.21, 1.22, and 1.23 clusters.
The vmware-k8s-1.21 variant included the packages needed to run a Kubernetes worker node as a VMware guest. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.21, 1.22, and 1.23 clusters.
The aws-k8s-1.22 variant included the packages needed to run a Kubernetes node in AWS. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.22, 1.23, and 1.24 clusters.
The aws-k8s-1.22-nvidia variant included the packages needed to run a Kubernetes node in AWS. It also included the required packages to configure containers to leverage NVIDIA GPUs. It supported self-hosted clusters and clusters managed by EKS.
This variant was compatible with Kubernetes 1.22, 1.23, and 1.24 clusters.
The metal-k8s-1.22 variant included the packages needed to run a Kubernetes node on bare metal. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.22, 1.23, and 1.24 clusters.
The vmware-k8s-1.22 variant included the packages needed to run a Kubernetes worker node as a VMware guest. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.22, 1.23, and 1.24 clusters.
The metal-k8s-1.23 variant included the packages needed to run a Kubernetes worker node on bare metal. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.23, 1.24, and 1.25 clusters.
The vmware-k8s-1.23 variant included the packages needed to run a Kubernetes worker node as a VMware guest. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.23, 1.24, and 1.25 clusters.
The vmware-k8s-1.24 variant included the packages needed to run a Kubernetes worker node as a VMware guest. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.24, 1.25, and 1.26 clusters.
The metal-k8s-1.24 variant included the packages needed to run a Kubernetes node on bare metal. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.24, 1.25, and 1.26 clusters.
The vmware-k8s-1.25 variant included the packages needed to run a Kubernetes worker node as a VMware guest. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.25, 1.26, 1.27, and 1.28 clusters.
The metal-k8s-1.25 variant included the packages needed to run a Kubernetes node on bare metal. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.25, 1.26, 1.27, and 1.28 clusters.
The vmware-k8s-1.26 variant included the packages needed to run a Kubernetes worker node as a VMware guest. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.26, 1.27, 1.28, and 1.29 clusters.
The metal-k8s-1.26 variant included the packages needed to run a Kubernetes node on bare metal. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.26, 1.27, 1.28, and 1.29 clusters.
The vmware-k8s-1.27 variant included the packages needed to run a Kubernetes worker node as a VMware guest. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.27, 1.28, 1.29, and 1.30 clusters.
The metal-k8s-1.27 variant included the packages needed to run a Kubernetes node on bare metal. It supported self-hosted clusters.
This variant was compatible with Kubernetes 1.27, 1.28, 1.29, and 1.30 clusters.
Say we want to create my-variant, a custom build of Bottlerocket that runs my-agent.
This listing shows the directory structure of our sample variant.
variants/my-variant
└── Cargo.toml
Each variant has a Cargo.toml file that lists the packages to install.
It also refers to a build.rs build script which tells Cargo to invoke our buildsys tool.
Artifacts for the variant are built as a side effect of Cargo running the script.
It points to /dev/null for the actual crate, since Cargo expects some Rust code to build, and is happy with an empty file.
Our sample variant has the following manifest.
[package]
name = "my-variant"
version = "0.1.0"
edition = "2018"
publish = false
build = "../build.rs"
[package.metadata.build-variant]
included-packages = [
"release",
"my-agent",
]
[package.metadata.build-variant.image-layout]
os-image-size-gib = 8
data-image-size-gib = 20
partition-plan = "unified"
[lib]
path = "../variants.rs"
[build-dependencies]
"my-agent" = { path = "../../packages/my-agent" }
"release" = { path = "../../packages/release" }The package.metadata table is ignored by Cargo and interpreted by our buildsys tool.
It contains an included-packages list which specifies the packages to install when building the image.
In the [build-dependencies] section, we specify the packages that need to be built, which is sometimes slightly different than included-packages.
This populates the Cargo build graph with all of the RPM packages that need to be built before the variant can be constructed.
Variants should almost always include the release package.
This pulls in the other core packages and includes essential configuration and services.
This variant includes the (optional) image-layout section, which allows the user to customize the layout of the image they are building.
os-image-size-gib is the size of the "OS" disk image in GiB.
data-image-size-gib is the size of the "data" disk image in GiB.
Though we've done so here for sake of demonstration, resizing the "data" disk image isn't necessary as it expands to fill the disk on boot.
partition-plan is the strategy used for image partitioning, with the options being "split" (the default) or "unified".
The "split" partition strategy has separate volumes for "OS" and "data", while "unified" has "OS" and "data" on a single volume.
See the documentation for the defaults and additional details.
Be sure to include publish = false for all packages, as these are not standard crates and should never appear on crates.io.
We reuse the same build script for all variants.
use std::process::{exit, Command};
fn main() -> Result<(), std::io::Error> {
let ret = Command::new("buildsys").arg("build-variant").status()?;
if !ret.success() {
exit(1);
}
Ok(())
}If you need a build script with different behavior, the recommended approach is to modify the buildsys tool.
The package.metadata table can be extended with declarative elements that enable the new feature.
To build your variant, run the following command in the top-level Bottlerocket directory.
cargo make -e BUILDSYS_VARIANT=my-variantThis will build all packages first, not just the ones needed by your variant.