Use a podTemplate inside the workspaceTemplate

[blampe]

We currently expect a fully-formed Pod spec, but we should use an apply configuration instead.

I have a branch where I started this work but I ran into a number of issues.

#713 (comment)

[alexstaeding]

I'll add some context for this from the slack conversation:

I was trying to add a pulumi plugin (for scala support) in my Stack custom resource for the operator. I found this issue/comment which suggests adding an init container to achieve this. (I assume Stack.spec.workspaceTemplate.spec.podTemplate.spec)

So far I've come across two issues with this approach, but I was just trying to get this to work relatively quickly so I may have missing something.

• The documentation mentions that containers "cannot be updated". The reality seems to be that a container must be specified, otherwise the stack spec is invalid e.g. The Stack "git" is invalid: spec.workspaceTemplate.spec.podTemplate.spec.containers: Required value. Specifying a dummy container works, and is added to the spec, although this should not be possible according to the docs - unless I misinterpreted "update". In any case, it certainly does not seem necessary.
• It seems that when the spec is merged, all entires that are merged put the elements of the patch spec first. This is a problem, because the initContainer for adding a plugin must run after the two default initContainers "bootstrap" and "fetch".

This is the quick & dirty spec I was trying to get to work:

workspaceTemplate:
    spec:
      podTemplate:
        spec:
          initContainers:
            - name: install-scala-cli-and-plugin
              image: curlimages/curl:latest
              command:
                - "/bin/sh"
                - "-c"
                - |
                  cd /share/workspace
                  curl -sSLf https://scala-cli.virtuslab.org/get | sh
                  pulumi plugin install language scala 0.3.2 --server github://api.github.com/VirtusLab/besom
          containers:
            - name: "dummy"
              image: busybox:latest

This is the resulting updated spec, which puts the custom initContainer before the others:

# ...
Pod Template:
  # ...
  Init Containers:
   install-scala-cli-and-plugin:
    Image:      curlimages/curl:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      cd /share/workspace
      curl -sSLf https://scala-cli.virtuslab.org/get | sh
      pulumi plugin install language scala 0.3.2 --server github://api.github.com/VirtusLab/besom
      
    Environment:  <none>
    Mounts:       <none>
   bootstrap:
    Image:      pulumi/pulumi-kubernetes-operator:v2.0.0-beta.3
    Port:       <none>
    Host Port:  <none>
    Args:
      cp
      /agent
      /tini
      /share/
    Environment:  <none>
    Mounts:
      /share from share (rw)
   fetch:
    Image:      pulumi/pulumi-kubernetes-operator:v2.0.0-beta.3
    Port:       <none>
    Host Port:  <none>
    Args:
      sh
      -c
      
      /share/agent init -t /share/source --git-url $GIT_URL --git-revision $GIT_REVISION &&
      ln -s /share/source/$GIT_DIR /share/workspace
          
    Environment:
      GIT_URL:       <...> # removed
      GIT_REVISION:  <...> # removed
      GIT_DIR:       
    Mounts:
      /share from share (rw)
  Containers:
   pulumi:
    Image:      docker.io/pulumi/pulumi:latest-nonroot
    Port:       50051/TCP
    Host Port:  0/TCP
    Command:
      /share/tini
      /share/agent
      --
      serve
      --workspace
      /share/workspace
      --skip-install
      --auth-mode
      kube
      --kube-workspace-namespace
      besom-start
      --kube-workspace-name
      git
    Requests:
      cpu:     100m
      memory:  64Mi
    Environment:
      POD_NAMESPACE:    (v1:metadata.namespace)
      POD_SA_NAME:      (v1:spec.serviceAccountName)
      AGENT_MEMLIMIT:  67108864 (requests.memory)
    Mounts:
      /share from share (rw)
   dummy:
    Image:        busybox:latest
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>