Stages not created AND automatic stages created

[agardnerIT]

This ticket is most likely a "misunderstanding" ticket from a newbie but showing my working / thinking here so it may inform further enhancements / docs.

I've onboarded project1 and I automatically see 3 stages: dev, hardening and production. I didn't apply these stages and they don't exist in my Git repo so the operator is creating them automatically. To my (extremely limited) understanding of GitOps, this is against the paradigm as my repo should be the source of truth and thus I now have stages that do not appear in my repo.

I believe this is done because creating a keptn project without stages is not currently possible.

Perhaps I'm forced down this workflow because I'm using GitHub.com and modifying one file at a time rather than committing everything at once. Even so, this is, IMO, how most beginners would probably approach things.

Nevertheless, I create a .keptn/stage.yaml file (side note, is this name hardcoded? I'd like to call this stages.yaml instead):

---
apiVersion: "keptn.sh/v1"
kind: "KeptnStage"
metadata:
  name: "foostage"
spec:
  project: "project1"

and my stage isn't created. In my Git upstream I see a shipyard.yaml file has been created for me so perhaps that is overriding the gitops operator:

apiVersion: "spec.keptn.sh/0.2.0"
kind: "Shipyard"
metadata:
  name: "podtato-head"
spec:
  stages:
    - name: "dev"
      sequences:
        - name: "dummy"
          tasks:
            - name: "dummy"
    - name: "hardening"
      sequences:
        - name: "dummy"
          tasks:
            - name: "dummy"
    - name: "production"
      sequences:
        - name: "dummy"
          tasks:
            - name: "dummy"

So I decide to delete the shipyard.yaml file from the Git upstream, which is OK right, since gitops will see take the repo as the source of truth and regenerate the shipyard file.

Other Issues

I'm (seemingly randomly) getting the below even though project1 does exist:

{
  "nextPageKey": "0",
  "projects": [{
      "creationDate": "1647649950814030400",
      "gitRemoteURI": "https://github.com/agardnerIT/keptn-gitops-project1",
      "gitUser": "agardnerit",
      "projectName": "project1",
      "shipyard": "apiVersion: \"spec.keptn.sh/0.2.0\"\nkind: \"Shipyard\"\nmetadata:\n  name: \"podtato-head\"\nspec:\n  stages:\n    - name: \"dev\"\n      sequences:\n        - name: \"dummy\"\n          tasks:\n            - name: \"dummy\"\n    - name: \"hardening\"\n      sequences:\n        - name: \"dummy\"\n          tasks:\n            - name: \"dummy\"\n    - name: \"production\"\n      sequences:\n        - name: \"dummy\"\n          tasks:\n            - name: \"dummy\"\n\n",
      "shipyardVersion": "spec.keptn.sh/0.2.0",
      "stages": [{
          "services": [],
          "stageName": "dev"
        }, {
          "services": [],
          "stageName": "hardening"
        }, {
          "services": [],
          "stageName": "production"
        }]
    }]
}

INFO    Reconciling Project     {"Request.Namespace": "keptn", "Request.Name": "project1"}
1.6476532207766266e+09  ERROR   Could not update status of project project1     {"Request.Namespace": "keptn", "Request.Name": "project1", "error": "keptnprojects.keptn.sh \"project1\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
1.6476532207767227e+09  ERROR   Finished Reconciling KeptnProject with error: %w        {"Request.Namespace": "keptn", "Request.Name": "project1", "error": "keptnprojects.keptn.sh \"project1\" not found"}
github.com/keptn-sandbox/keptn-gitops-operator/keptn-operator/controllers/keptnprojectcontroller.(*KeptnProjectReconciler).Reconcile
        /workspace/controllers/keptnprojectcontroller/keptnproject_controller.go:153
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
1.6476532207767754e+09  ERROR   controller.keptnproject Reconciler error        {"reconciler group": "keptn.sh", "reconciler kind": "KeptnProject", "name": "project1", "namespace": "keptn", "error": "keptnprojects.keptn.sh \"project1\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

This thread may lead to Keptn RFEs but the workflow that would work for me (and IMHO is the "most logical" esp. to a new user is):

1. Install operators
2. Create `instance.yaml`
3. Create `project.yaml` (this is an empty project without stages)
4. Create `stages.yaml` (creates an empty stage without sequences)
5. Create `sequences.yaml` with one or more sequences and tasks
6. Modify `stages.yaml` to link sequences

Up to step 4 would be the equivalent of the existing:

keptn create project project2 --shipyard=shipyard.yaml

Where shipyard.yaml:

apiVersion: "spec.keptn.sh/0.2.0"
kind: "Shipyard"
metadata:
name: "project2-shipyard"
spec:
stages:
- name: "stage1"

[thschue]

Hello!

At the moment you can't create create or remove stages, as keptn itself does not have this functionality (see keptn/enhancement-proposals#70). Therefore, initially, 3 Stages are created by default. This can also be overridden (base64-encode a shipyard file and paste this to spec.initialShipyard) of a KeptnProject.

Will take a look on the other errors and warnings later ...