Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(trait): disable incremental image option #4816

Merged
merged 2 commits into from
Oct 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions config/crd/bases/camel.apache.org_integrationkits.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,10 @@ spec:
description: Can be used to enable or disable a trait. All
traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to reuse
existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use TasksRequestCPU
Expand Down
8 changes: 8 additions & 0 deletions config/crd/bases/camel.apache.org_integrationplatforms.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,10 @@ spec:
description: Can be used to enable or disable a trait. All
traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to reuse
existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use TasksRequestCPU
Expand Down Expand Up @@ -2328,6 +2332,10 @@ spec:
description: Can be used to enable or disable a trait. All
traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to reuse
existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use TasksRequestCPU
Expand Down
4 changes: 4 additions & 0 deletions config/crd/bases/camel.apache.org_integrations.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6423,6 +6423,10 @@ spec:
description: Can be used to enable or disable a trait. All
traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to reuse
existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use TasksRequestCPU
Expand Down
4 changes: 4 additions & 0 deletions config/crd/bases/camel.apache.org_kameletbindings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6705,6 +6705,10 @@ spec:
description: Can be used to enable or disable a trait.
All traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to
reuse existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use
Expand Down
4 changes: 4 additions & 0 deletions config/crd/bases/camel.apache.org_pipes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6703,6 +6703,10 @@ spec:
description: Can be used to enable or disable a trait.
All traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to
reuse existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use
Expand Down
4 changes: 3 additions & 1 deletion docs/modules/ROOT/pages/architecture/incremental-image.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,6 @@ In order to reduce the time to build and execute an application we adopt various

If you're familiar with Camel K architecture, you know that an Integration uses an IntegrationKit, which is, a reusable resource containing all the dependencies and capabilities required to run a certain "class" of Camel applications. The IntegrationKit is bound to a container image stored in a registry.

When the IntegrationKit is created, it uses as a base image a JDK based container and add certain applications layers on top of it. However, instead of using always the same root base image, we check if an IntegrationKit with a subset of dependencies already exists, sparing quite some time on the generation of the new container image.
When the IntegrationKit is created, it uses as a base image a JDK based container and add certain applications layers on top of it. However, instead of using always the same root base image, we check if an IntegrationKit with a subset of dependencies already exists, sparing quite some time on the generation of the new container image.

NOTE: you may disable the incremental image feature and always build and package your Camel application from scratch using builder trait option `-t builder.incremental-image-build=false`.
7 changes: 7 additions & 0 deletions docs/modules/ROOT/partials/apis/camel-k-crds.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5789,6 +5789,13 @@ string

The strategy to use, either `pod` or `routine` (default routine)

|`incrementalImageBuild` +
bool
|


Use the incremental image build option, to reuse existing containers (default `true`)

|`orderStrategy` +
string
|
Expand Down
4 changes: 4 additions & 0 deletions docs/modules/traits/pages/builder.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@ The following configuration options are available:
| string
| The strategy to use, either `pod` or `routine` (default routine)

| builder.incremental-image-build
| bool
| Use the incremental image build option, to reuse existing containers (default `true`)

| builder.order-strategy
| string
| The build order strategy to use, either `dependencies`, `fifo` or `sequential` (default sequential)
Expand Down
48 changes: 48 additions & 0 deletions e2e/commonwithcustominstall/incremental_build_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -137,3 +137,51 @@ func TestRunIncrementalBuildPod(t *testing.T) {
Expect(Kamel("delete", "--all", "-n", ns).Execute()).To(Succeed())
})
}

func TestRunIncrementalBuildOff(t *testing.T) {
WithNewTestNamespace(t, func(ns string) {
operatorID := "camel-k-standard-build"
Expect(KamelInstallWithID(operatorID, ns).Execute()).To(Succeed())

name := "java"
Expect(KamelRunWithID(operatorID, ns, "files/Java.java",
"--name", name,
).Execute()).To(Succeed())
Eventually(IntegrationPodPhase(ns, name), TestTimeoutLong).Should(Equal(corev1.PodRunning))
Eventually(IntegrationConditionStatus(ns, name, v1.IntegrationConditionReady), TestTimeoutShort).Should(Equal(corev1.ConditionTrue))
Eventually(IntegrationLogs(ns, name), TestTimeoutShort).Should(ContainSubstring("Magicstring!"))
integrationKitName := IntegrationKit(ns, name)()
Eventually(Kit(ns, integrationKitName)().Status.BaseImage).Should(Equal(defaults.BaseImage()))

t.Run("Don't reuse previous kit", func(t *testing.T) {
nameClone := "java-clone"
Expect(KamelRunWithID(operatorID, ns, "files/Java.java",
"--name", nameClone,
"-t", "builder.incremental-image-build=false",
).Execute()).To(Succeed())
Eventually(IntegrationPodPhase(ns, nameClone), TestTimeoutLong).Should(Equal(corev1.PodRunning))
Eventually(IntegrationConditionStatus(ns, nameClone, v1.IntegrationConditionReady), TestTimeoutShort).Should(Equal(corev1.ConditionTrue))
Eventually(IntegrationLogs(ns, nameClone), TestTimeoutShort).Should(ContainSubstring("Magicstring!"))
integrationCloneKitName := IntegrationKit(ns, nameClone)()
Eventually(Kit(ns, integrationCloneKitName)().Status.BaseImage).Should(Equal(defaults.BaseImage()))
})

t.Run("Don't create incremental kit", func(t *testing.T) {
// Another integration that should be built on top of the previous IntegrationKit
// just add a new random dependency
nameIncremental := "java-incremental"
Expect(KamelRunWithID(operatorID, ns, "files/Java.java",
"--name", nameIncremental,
"-d", "camel:zipfile",
"-t", "builder.incremental-image-build=false",
).Execute()).To(Succeed())
Eventually(IntegrationPodPhase(ns, nameIncremental), TestTimeoutLong).Should(Equal(corev1.PodRunning))
Eventually(IntegrationConditionStatus(ns, nameIncremental, v1.IntegrationConditionReady), TestTimeoutShort).Should(Equal(corev1.ConditionTrue))
Eventually(IntegrationLogs(ns, nameIncremental), TestTimeoutShort).Should(ContainSubstring("Magicstring!"))
integrationIncrementalKitName := IntegrationKit(ns, nameIncremental)()
Eventually(Kit(ns, integrationIncrementalKitName)().Status.BaseImage).Should(Equal(defaults.BaseImage()))
})

Expect(Kamel("delete", "--all", "-n", ns).Execute()).To(Succeed())
})
}
4 changes: 4 additions & 0 deletions helm/camel-k/crds/crd-integration-kit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,10 @@ spec:
description: Can be used to enable or disable a trait. All
traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to reuse
existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use TasksRequestCPU
Expand Down
8 changes: 8 additions & 0 deletions helm/camel-k/crds/crd-integration-platform.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,10 @@ spec:
description: Can be used to enable or disable a trait. All
traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to reuse
existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use TasksRequestCPU
Expand Down Expand Up @@ -2328,6 +2332,10 @@ spec:
description: Can be used to enable or disable a trait. All
traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to reuse
existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use TasksRequestCPU
Expand Down
4 changes: 4 additions & 0 deletions helm/camel-k/crds/crd-integration.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6423,6 +6423,10 @@ spec:
description: Can be used to enable or disable a trait. All
traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to reuse
existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use TasksRequestCPU
Expand Down
4 changes: 4 additions & 0 deletions helm/camel-k/crds/crd-kamelet-binding.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6705,6 +6705,10 @@ spec:
description: Can be used to enable or disable a trait.
All traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to
reuse existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use
Expand Down
4 changes: 4 additions & 0 deletions helm/camel-k/crds/crd-pipe.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6703,6 +6703,10 @@ spec:
description: Can be used to enable or disable a trait.
All traits share this common property.
type: boolean
incrementalImageBuild:
description: Use the incremental image build option, to
reuse existing containers (default `true`)
type: boolean
limitCPU:
description: 'When using `pod` strategy, the maximum amount
of CPU required by the pod builder. Deprecated: use
Expand Down
2 changes: 2 additions & 0 deletions pkg/apis/camel/v1/trait/builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ type BuilderTrait struct {
Properties []string `property:"properties" json:"properties,omitempty"`
// The strategy to use, either `pod` or `routine` (default routine)
Strategy string `property:"strategy" json:"strategy,omitempty"`
// Use the incremental image build option, to reuse existing containers (default `true`)
IncrementalImageBuild *bool `property:"incremental-image-build" json:"incrementalImageBuild,omitempty"`
// The build order strategy to use, either `dependencies`, `fifo` or `sequential` (default sequential)
OrderStrategy string `property:"order-strategy" json:"orderStrategy,omitempty"`
// When using `pod` strategy, the minimum amount of CPU required by the pod builder.
Expand Down
5 changes: 5 additions & 0 deletions pkg/apis/camel/v1/trait/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions pkg/builder/image.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,11 @@ import (
)

const (
ContextDir = "context"
DeploymentDir = "/deployments"
// ContextDir is the directory used to package the container.
ContextDir = "context"
// DeploymentDir is the directory used in the runtime application to deploy the artifacts.
DeploymentDir = "/deployments"
// DependenciesDir is the directory used to store required dependencies.
DependenciesDir = "dependencies"
)

Expand Down Expand Up @@ -136,9 +139,6 @@ func incrementalImageContext(ctx *builderContext) error {
ctx.SelectedArtifacts = append(ctx.SelectedArtifacts, entry)
}
}
} else if ctx.BaseImage == "" {
// TODO: transient workaround to be removed in 1.8.x
ctx.BaseImage = defaults.BaseImage()
}

return nil
Expand Down
6 changes: 5 additions & 1 deletion pkg/trait/builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ import (
"github.com/apache/camel-k/v2/pkg/util/property"
)

const (
builderTraitID = "builder"
)

var commandsRegexp = regexp.MustCompile(`"[^"]+"|[\w/-]+`)

type builderTrait struct {
Expand All @@ -43,7 +47,7 @@ type builderTrait struct {

func newBuilderTrait() Trait {
return &builderTrait{
BaseTrait: NewBaseTrait("builder", 600),
BaseTrait: NewBaseTrait(builderTraitID, 600),
}
}

Expand Down
17 changes: 16 additions & 1 deletion pkg/trait/quarkus.go
Original file line number Diff line number Diff line change
Expand Up @@ -366,7 +366,11 @@ func (t *quarkusTrait) applyWhenBuildSubmitted(e *Environment) error {
// Default, if nothing is specified
buildTask.Maven.Properties["quarkus.package.type"] = string(fastJarPackageType)
packageSteps = append(packageSteps, builder.Quarkus.ComputeQuarkusDependencies)
packageSteps = append(packageSteps, builder.Image.IncrementalImageContext)
if t.isIncrementalImageBuild(e) {
packageSteps = append(packageSteps, builder.Image.IncrementalImageContext)
} else {
packageSteps = append(packageSteps, builder.Image.StandardImageContext)
}
// Create the dockerfile, regardless it's later used or not by the publish strategy
packageSteps = append(packageSteps, builder.Image.JvmDockerfile)
}
Expand Down Expand Up @@ -396,6 +400,17 @@ func (t *quarkusTrait) isNativeKit(e *Environment) (bool, error) {
}
}

func (t *quarkusTrait) isIncrementalImageBuild(e *Environment) bool {
// We need to get this information from the builder trait
if trait := e.Catalog.GetTrait(builderTraitID); trait != nil {
builder, ok := trait.(*builderTrait)
return ok && pointer.BoolDeref(builder.IncrementalImageBuild, true)
}

// Default always to true for performance reasons
return true
}

func (t *quarkusTrait) applyWhenKitReady(e *Environment) error {
if e.IntegrationInRunningPhases() && t.isNativeIntegration(e) {
container := e.GetIntegrationContainer()
Expand Down
4 changes: 3 additions & 1 deletion pkg/trait/quarkus_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
v1 "github.com/apache/camel-k/v2/pkg/apis/camel/v1"
"github.com/apache/camel-k/v2/pkg/builder"
"github.com/apache/camel-k/v2/pkg/util/camel"
"github.com/apache/camel-k/v2/pkg/util/test"
)

func TestConfigureQuarkusTraitBuildSubmitted(t *testing.T) {
Expand Down Expand Up @@ -95,8 +96,9 @@ func TestApplyQuarkusTraitAnnotationKitConfiguration(t *testing.T) {
func createNominalQuarkusTest() (*quarkusTrait, *Environment) {
trait, _ := newQuarkusTrait().(*quarkusTrait)
trait.Enabled = pointer.Bool(true)

client, _ := test.NewFakeClient()
environment := &Environment{
Catalog: NewCatalog(client),
CamelCatalog: &camel.RuntimeCatalog{},
Integration: &v1.Integration{
Spec: v1.IntegrationSpec{
Expand Down
4 changes: 4 additions & 0 deletions resources/traits.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,10 @@ traits:
- name: strategy
type: string
description: The strategy to use, either `pod` or `routine` (default routine)
- name: incremental-image-build
type: bool
description: Use the incremental image build option, to reuse existing containers
(default `true`)
- name: order-strategy
type: string
description: The build order strategy to use, either `dependencies`, `fifo` or
Expand Down