pipeline: How to optimize assets bundling

[rverma-dev]

Describe the bug

Considering a simple pipeline creation with just cluster builder

export class WorkshopPipelineStage extends cdk.Stage {
  constructor(scope: Construct, id: string, props?: cdk.StageProps) {
    super(scope, id, props);

    const KMS_RESOURCE = "kms-key-22";
    const base = bp.EksBlueprint.builder()
      .account(props!.env!.account)
      .region(props!.env!.region)
      .resourceProvider(
        bp.GlobalResources.Vpc,
        new bp.VpcProvider(undefined, "100.64.0.0/16", [
          "100.64.0.0/24",
          "100.64.1.0/24",
          "100.64.2.0/24",
        ])
      )
      .resourceProvider(KMS_RESOURCE, {
        provide(context): cdk.aws_kms.Key {
          return new kms.Key(context.scope, KMS_RESOURCE);
        },
      });

    const kmsKey: kms.Key = bp.getNamedResource(KMS_RESOURCE);
    const builder = () => base.clone();
    const publicCluster = {
      version: KubernetesVersion.V1_25,
      vpcSubnets: [{ subnetType: ec2.SubnetType.PUBLIC }],
    };
    builder()
      .clusterProvider(new bp.FargateClusterProvider(publicCluster))
      .build(this, "fargate-blueprint");
  }
}

export class WorkloadPipelineStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const pipeline = new pipelines.CodePipeline(this, "Pipeline", {
      pipelineName: "WorkshopPipeline",
      synth: new pipelines.CodeBuildStep("SynthStep", {
        input: pipelines.CodePipelineSource.gitHub(
          "rverma-nsl/demo-pipeline",
          "main"
        ),
        installCommands: ["npm install -g aws-cdk"],
        commands: ["npm ci", "npm run build", "npx cdk synth"],
      }),
    });
    const deploy = new WorkshopPipelineStage(this, "Deploy", props);
    const deployStage = pipeline.addStage(deploy);
  }
}

The resulting pipeline creates additional 9 codebuild projects such as

PipelineAssetsFileAsset81DAB433B:
    Type: AWS::CodeBuild::Project
    Properties:
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/standard:6.0
        ImagePullCredentialsType: CODEBUILD
        PrivilegedMode: false
        Type: LINUX_CONTAINER
      ServiceRole:
        Fn::GetAtt:
          - PipelineAssetsFileRole59943A77
          - Arn
      Source:
        BuildSpec: |-
          {
            "version": "0.2",
            "phases": {
              "install": {
                "commands": [
                  "npm install -g cdk-assets@2"
                ]
              },
              "build": {
                "commands": [
                  "cdk-assets --path \"assembly-DemoStack-Deploy/DemoStackDeployfargateblueprint35D9E874.assets.json\" --verbose publish \"34f979cb5bd0dd0be7978a0d34a422b5e76b3ed24adbaac73507c4b51053ecb9:415505189627-ap-south-1\""
                ]
              }
            }
          }
        Type: CODEPIPELINE
      Cache:
        Type: NO_CACHE
      Description: Pipeline step DemoStack/Pipeline/Assets/FileAsset8
      EncryptionKey: alias/aws/s3
    Metadata:
      aws:cdk:path: DemoStack/Pipeline/Assets/FileAsset8/Resource

This inflates the number of codebuild projects, complicates the end-user understanding of what happens in those assets and there is no way to control if the assets can be stored over s3. We are provisioning a lot of eks-clusters and wondering if this way is good.

Contrary if we provision the cluster directly, no assets are created at all

const cluster = new eks.Cluster(this, 'demogo-cluster', {
      clusterName: 'demo',
      version: eks.KubernetesVersion.V1_21,
      mastersRole: clusterAdmin,
      defaultCapacity: 2
    });

    cluster.addNodegroupCapacity('spot-ng', {
      instanceTypes: [
        new ec2.InstanceType('m5.large'),
        new ec2.InstanceType('m5a.large')
      ],
      minSize: 2,
      capacityType: eks.CapacityType.SPOT
    })

Expected Behavior

1. No of assets has to optimize
2. Control the output of assets to atleast s3
3. Some sort of asset bundling

Current Behavior

The asset creation and handling is like blackbox

Reproduction Steps

Provided as part of issue discription

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.76.0

EKS Blueprints Version

1.70.0

Node.js Version

18.16

Environment details (OS name and version, etc.)

Macos arm

Other information

No response