Apply package config from meta-buildpack folder in pack build

[colincasey]

Summary

When supplying a meta-buildpack folder as an argument to the build command (e.g.; pack build --buildpack <path to meta-buildpack folder> ...) the package.toml configuration file within the directory is ignored. If the meta-buildpack references a buildpack that only exists locally, that buildpack will fail to resolve.

This behavior differs from using a meta-buildpack with pack buildpack package ... and then pack build --buildpack <CNB file or OCI image reference> .... The CNB file or OCI image reference created contains all the referenced buildpacks listed from the package.toml within itself and those get extracted and included as extra buildpacks during the build process.

This PR attempts to detect when a meta-buildpack folder is supplied during the build process and, if a package.toml file is found, it will extract any referenced buildpacks so that there is no difference in behavior when supplying a meta-buildpack as a folder, CNB file, or OCI image.

Output

Before

Builder 'localhost:32801/test/builder-yyxwibmzox' is trusted
Warning: Ignoring the provided lifecycle image as the builder is trusted, running the creator in a single container using the provided builder
Pulling image 'localhost:32801/test/builder-yyxwibmzox:latest'
latest: Pulling from test/builder-yyxwibmzox
Digest: sha256:94acac98005e62b5f355b8e29c04ffba93b0a615789498e1753147c80616cc4b
Status: Image is up to date for localhost:32801/test/builder-yyxwibmzox:latest
Selected run image mirror 'localhost:32801/pack-test/run'
Pulling image 'localhost:32801/pack-test/run'
latest: Pulling from pack-test/run
Digest: sha256:a0170a81c1d9e1c37c2f225237faf994cd46fdda4e5efa6461884352c7a2f6fa
Status: Image is up to date for localhost:32801/pack-test/run:latest
Downloading buildpack from URI: 'file:///var/folders/r4/xq3q_hjj2v1ghqlnr1k1zrdw0000gp/T/folder-buildpack-tests-3046813775/meta-buildpack'
Adding buildpack 'local/meta-bp' version 'local-meta-bp-version' to builder
Setting custom order
Creating builder with the following buildpacks:
-> 'read/env@read-env-version'
-> '[email protected]'
-> '[email protected]'
-> 'simple/layers@simple-layers-version'
-> 'local/meta-bp@local-meta-bp-version'
Adding buildpack 'local/meta-bp@local-meta-bp-version' (diffID=sha256:a0d2785b194a2d99d737bd7dae51e680d7eb84301896b9b52de8e3475bcf07b5)
Using build cache volume 'pack-cache-some-org_xnjpgfwzeu_latest-2785214a39ea.build'
Running the 'creator' on OS 'linux' with:
Container Settings:
 Args: '/cnb/lifecycle/creator -daemon -launch-cache /launch-cache -log-level debug -app /workspace -cache-dir /cache -run-image localhost:32801/pack-test/run localhost:32801/some-org/xnjpgfwzeu'
 System Envs: 'CNB_PLATFORM_API=0.12'
 Image: 'pack.local/builder/77637273677469766572:latest'
 User: 'root'
 Labels: 'map[author:pack]'
Host Settings:
 Binds: 'pack-cache-some-org_xnjpgfwzeu_latest-2785214a39ea.build:/cache /var/run/docker.sock:/var/run/docker.sock pack-cache-some-org_xnjpgfwzeu_latest-2785214a39ea.launch:/launch-cache pack-layers-ebokvymwaa:/layers pack-app-emnmmwelif:/workspace'
 Network Mode: ''
Starting creator...
Parsing inputs...
Ensuring privileges...
Executing command...
===> ANALYZING
Image with name "localhost:32801/some-org/xnjpgfwzeu" not found
Found image with identifier "0081b3f2f99ac10ecff9af8491ae82ea9714e716236498722e84be0c8aeb523a"
===> DETECTING
Warning: Buildpack 'local/meta-bp@local-meta-bp-version' requests deprecated API '0.2'
ERROR: failed to detect: open /cnb/buildpacks/local_meta-bp-dep/local-meta-bp-version/buildpack.toml: no such file or directory
ERROR: failed to build: executing lifecycle: failed with status code: 22

After

Builder 'localhost:32800/test/builder-kdzflawfxw' is trusted
Warning: Ignoring the provided lifecycle image as the builder is trusted, running the creator in a single container using the provided builder
Pulling image 'localhost:32800/test/builder-kdzflawfxw:latest'
latest: Pulling from test/builder-kdzflawfxw
Digest: sha256:67e70168f783b48325bf46b4fc3c4898c1605425c2ec2cef49c9a736189011af
Status: Image is up to date for localhost:32800/test/builder-kdzflawfxw:latest
Selected run image mirror 'localhost:32800/pack-test/run'
Pulling image 'localhost:32800/pack-test/run'
latest: Pulling from pack-test/run
Digest: sha256:a0170a81c1d9e1c37c2f225237faf994cd46fdda4e5efa6461884352c7a2f6fa
Status: Image is up to date for localhost:32800/pack-test/run:latest
Downloading buildpack from URI: 'file:///var/folders/r4/xq3q_hjj2v1ghqlnr1k1zrdw0000gp/T/folder-buildpack-tests-3862156550/meta-buildpack'
Downloading buildpack from URI: 'file:///var/folders/r4/xq3q_hjj2v1ghqlnr1k1zrdw0000gp/T/folder-buildpack-tests-3862156550/meta-buildpack-dependency'
Adding buildpack 'local/meta-bp' version 'local-meta-bp-version' to builder
Adding buildpack 'local/meta-bp-dep' version 'local-meta-bp-version' to builder
Setting custom order
Creating builder with the following buildpacks:
-> 'read/env@read-env-version'
-> '[email protected]'
-> '[email protected]'
-> 'simple/layers@simple-layers-version'
-> 'local/meta-bp@local-meta-bp-version'
-> 'local/meta-bp-dep@local-meta-bp-version'
Adding buildpack 'local/meta-bp-dep@local-meta-bp-version' (diffID=sha256:475f9c8b8b532f89d152006dad53c88c45cf6fc3cc9f102deb886853ae4c7a65)
Adding buildpack 'local/meta-bp@local-meta-bp-version' (diffID=sha256:a0d2785b194a2d99d737bd7dae51e680d7eb84301896b9b52de8e3475bcf07b5)
Using build cache volume 'pack-cache-some-org_fmnponjbum_latest-c326c3a5e2b0.build'
Running the 'creator' on OS 'linux' with:
Container Settings:
  Args: '/cnb/lifecycle/creator -daemon -launch-cache /launch-cache -log-level debug -app /workspace -cache-dir /cache -run-image localhost:32800/pack-test/run localhost:32800/some-org/fmnponjbum'
  System Envs: 'CNB_PLATFORM_API=0.12'
  Image: 'pack.local/builder/6d61687075626a6b7a74:latest'
  User: 'root'
  Labels: 'map[author:pack]'
Host Settings:
  Binds: 'pack-cache-some-org_fmnponjbum_latest-c326c3a5e2b0.build:/cache /var/run/docker.sock:/var/run/docker.sock pack-cache-some-org_fmnponjbum_latest-c326c3a5e2b0.launch:/launch-cache pack-layers-pgzuyrusfr:/layers pack-app-aibgffsssl:/workspace'
  Network Mode: ''
Starting creator...
Parsing inputs...
Ensuring privileges...
Executing command...
===> ANALYZING
Image with name "localhost:32800/some-org/fmnponjbum" not found
Found image with identifier "0081b3f2f99ac10ecff9af8491ae82ea9714e716236498722e84be0c8aeb523a"
===> DETECTING
Warning: Buildpack 'local/meta-bp@local-meta-bp-version' requests deprecated API '0.2'
Checking for match against descriptor: {windows *  []}
Checking for match against descriptor: {linux *  []}
======== Results ========
pass: local/meta-bp-dep@local-meta-bp-version
Resolving plan... (try #1)
local/meta-bp-dep local-meta-bp-version
===> RESTORING
Reading buildpack directory: /layers/local_meta-bp-dep
Reading buildpack directory: /layers/local_meta-bp-dep
===> BUILDING
Starting build
Running build for buildpack local/meta-bp-dep@local-meta-bp-version
Looking up buildpack
Finding plan
Updating plan entries
Creating plan directory
Preparing paths
Running build command
---> Build: Local Meta-Buildpack Dependency
Processing layers
Updating environment
Reading output files
Updating buildpack processes
Updating process list
Finished running build for buildpack local/meta-bp-dep@local-meta-bp-version
Copying SBOM files
Creating SBOM files for legacy BOM
Listing processes
Finished build
===> EXPORTING
Warning: no analyzed metadata found at path '/layers/analyzed.toml'
Copying SBOM lifecycle.sbom.cdx.json to /layers/sbom/build/buildpacksio_lifecycle/sbom.cdx.json
Copying SBOM lifecycle.sbom.spdx.json to /layers/sbom/build/buildpacksio_lifecycle/sbom.spdx.json
Copying SBOM lifecycle.sbom.syft.json to /layers/sbom/build/buildpacksio_lifecycle/sbom.syft.json
Copying SBOM launcher.sbom.cdx.json to /layers/sbom/launch/buildpacksio_lifecycle/launcher/sbom.cdx.json
Copying SBOM launcher.sbom.spdx.json to /layers/sbom/launch/buildpacksio_lifecycle/launcher/sbom.spdx.json
Copying SBOM launcher.sbom.syft.json to /layers/sbom/launch/buildpacksio_lifecycle/launcher/sbom.syft.json
Reading buildpack directory: /layers/local_meta-bp-dep
Processing buildpack directory: /layers/local_meta-bp-dep
Found SBOM of type launch for at /layers/sbom/launch
Reusing tarball for layer "buildpacksio/lifecycle:launch.sbom" with SHA: sha256:56034527a5b950782c3736cc281c75a07bc9fbc9d92c03ccc40cb652acf3e95d
Adding layer 'buildpacksio/lifecycle:launch.sbom'
Layer 'buildpacksio/lifecycle:launch.sbom' SHA: sha256:56034527a5b950782c3736cc281c75a07bc9fbc9d92c03ccc40cb652acf3e95d
Layer 'slice-1' SHA: sha256:cd68344da40adc783009aec5ecdf1d3f05efb5de69284d7eca5dbe4dfe50e006
Adding 1/1 app layer(s)
Reusing tarball for layer "buildpacksio/lifecycle:launcher" with SHA: sha256:35268043f3963e7c73ea53a594924bfe681adda9a2a0ed0201811034d7593aef
Adding layer 'buildpacksio/lifecycle:launcher'
Layer 'buildpacksio/lifecycle:launcher' SHA: sha256:35268043f3963e7c73ea53a594924bfe681adda9a2a0ed0201811034d7593aef
Reusing tarball for layer "buildpacksio/lifecycle:config" with SHA: sha256:1a9827e598c2d0ae30012e928ca820555a95fbd62f7f3d0f3705b0a9bf99c3df
Adding layer 'buildpacksio/lifecycle:config'
Layer 'buildpacksio/lifecycle:config' SHA: sha256:1a9827e598c2d0ae30012e928ca820555a95fbd62f7f3d0f3705b0a9bf99c3df
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting CNB_LAYERS_DIR=/layers
Setting CNB_APP_DIR=/workspace
Setting CNB_PLATFORM_API=0.12
Setting CNB_DEPRECATION_MODE=quiet
Prepending /cnb/process and /cnb/lifecycle to PATH
Setting WORKDIR: '/workspace'
no default process type
Setting ENTRYPOINT: '/cnb/lifecycle/launcher'
Saving localhost:32800/some-org/fmnponjbum...
*** Images (160454a0198a):
      localhost:32800/some-org/fmnponjbum

*** Image ID: 160454a0198a5285ec5d38f21623717cd7c42430b9ce1ae5d2b437249713ef26
Reading buildpack directory: /layers/local_meta-bp-dep
Successfully built image 'localhost:32800/some-org/fmnponjbum'

• Should this change be documented?
  • Yes, see #___
  • No

Resolves #1320

[natalieparellano]

Tests are failing due to moby/moby#45935

[dlion]

Thank you for this PR @colincasey it looks almost good to me, I left some comments 😄

[natalieparellano]

Thanks for the PR @colincasey! This looks good to me. I liked the small readability suggestion from @dlion ❤️