Feedback for the profiles catalog structure and content

[darrylweaver]

Please can anyone in CX give us feedback on the structure that we decided on for the profiles catalog and if this makes sense to you.

Also please include any comments that you have on the content included so far.

[darrylweaver]

We are aware that there will indeed be a separation of concerns by team.
We should probably consider splitting up profiles to reflect the different teams concerns.

For example, most broadly there can be a platform team and an application team.
The platform team would be concerned about observability components, e.g. kube-prometheus-stack.
The app team would be concerned about having a database component, e.g. postgres operator.

[steve-fraser]

Including a database profiles could prove difficult if there are monitoring dependency requirements. For example which would cause a double install of the observability profile

[fire-ant]

If we are even considering recommending running production data stores on K8s I'd suggest to run right-sized/optimised nodes in dedicated cluster(s). That removes the monitoring duplication (if I understand the statement) but implies that there may be infra preferences and overhead. That may not suit everyone but being conservative around customer data is pragmatic

[Nalum]

In terms of the double install, should pctl be updated to build up a deployment tree?

Once it has a picture of what it is doing it could then see there are two/three profiles that are setting up Prometheus and act on that. This act could be to prompt for confirmation or merge those two installs into a single install.

Would it make sense, rather than have multiple profiles install Prometheus have one that installs the operator and then profiles that rely on Prometheus use that operator?

I suppose this gets into the dependency issues, but if we know the observability sub-profile has this operator setup and is standard to all gitops-enterprise-* profiles maybe it's not much of an issue.

[souleb]

How strong do we feel about having istio and flagger in the base profile (I have missed the conversation regarding that, if so sorry). Many companies find istio complicated to maintain. And progressive delivery is too soon for many as well. Is it the right approach to force them to use it in order to start with profiles?

[steve-fraser]

How strong do we feel about having istio and flagger in the base profile (I have missed the conversation regarding that, if so sorry). Many companies find istio complicated to maintain. And progressive delivery is too soon for many as well. Is it the right approach to force them to use it in order to start with profiles?

There really hasn't been a conversation about it yet. I included it in base to make the installation easier for the customer considering we are including a service-mesh it may increase overall ease of adoption. I also included it thinking about if we wanted to allow automatic updates of let's say our agent image (for maybe non-production environments). When I was managing software, I loved having a trusted software vendor that allowed this feature to reduce my overall people cost.

However, there could be a bit too much of customization there if the customers typically use a lot of different types of datasources.

[darrylweaver]

I guess we should acknowledge that Istio can be a big ask to maintain and understand, so we should have simpler options available too.

[fire-ant]

theres security overhead to think about if your not using it and resource mgmt and optimisation if you are (that goes for both). I dont even see customers who aren't doing multi-region at a minimum being interested in istio, which would preclude it from the majority of deployments. Nice to haves but not must haves (and there are better solutions for mTLS more suited to vanilla deployments, which I dont think I'd suggest for base either)

[steve-fraser]

@darrylweaver and @fire-ant we reviewed this on the ref arch call today. The current thinking is that we change out istio in the base profile to a standard Nginx ingress controller. We can still use Flagger in that case if we want to in the base profile.

[Nalum]

Bringing in a slack thread about the JMX exporter, do we need further discussion on exporters in general?

@Nalum 23 hours ago
Should we add Jaeger to the observability profile or add a tracing profile with it?

@pfcurtis 22 hours ago
I would say keep Jaeger separate from the "traditional" observability (prom+grafana) as it is more APM?

@Nalum 21 hours ago
👍 PR setup #54

@pfcurtis 21 hours ago
There's probably a couple more application debugging tools that would fit into the same category. A weird example: the JMX exporter for Prometheus

@Nalum 21 hours ago
I'm not familiar with JMX Exporter, is that deployed one for each target or does one instance handle multiple targets? My initial thoughts on exported would be that they are deployed alongside the application they are exporting for.

@steve-fraser 18 hours ago
My thoughts are to keep in the observability profile and let the customer decided to use it or not with it. Considering that if the JMX exporter requires promtheus is will cause a double install if not included the observability

@steve-fraser 18 hours ago
if we don't included Prometheus in the installation we could be shipping a half baked profile (edited)

@steve-fraser 18 hours ago
@pfcurtis ^ thoughts

@pfcurtis 17 hours ago
IMHO ... (take with a grain of salt) Any customer that is doing APM with tools like JMX would either already have Prom installed or their JMX is configured to go elsewhere (eg. Datadog).
I would say that if someone attempts to install a profile for APM, like the JMX exporter, that it would fail or warn if Prom isn't already there.
The same is true for LinkerD. Their viz install brings in Prom, but it can configured to talk to an existing Prom. Do we not install, or do we install even when there are duplicative services?
A question for the ages ...

[fire-ant]

Profile Candidate: StorageOS
OSS technology: StorageOS
Parent Company: https://www.ondat.io/
CNCF member: yes
project level: N/A

generic install:
https://github.com/storageos/charts/tree/master/stable/storageos-operator#installing-the-chart
/enterprise:
https://docs.ondat.io/docs/install/kubernetes/
use cases:
https://github.com/storageos/use-cases

profile operators:
https://github.com/elastic/cloud-on-k8s
https://github.com/influxdata/influxdata-operator
https://github.com/grafana-operator/grafana-operator

Preface:

As per #46 and Mark Ramms novel questioning around tenanting storage backends for single clusters I think we should harbour opinions around data/stateful workload(s) and provide a broad base with a view that there is an increasingly higher likelihood that we need to accommodate customers who want to put their data in kubernetes.

StorageOS is likely a top three candidate to use as a backend/base for stateful components and dedicated stateful workload clusters. As StorageOS are now moving to an enterprise support model (see Ondat) where they can serve numerous customer needs through a formal engagement model. This is a good thing and has healthy benefits for purveyors of their platform as well as users (shared support etc).

User story:

As a Platform engineer tasked with transforming our on-premise stateful workloads I need a cloud native data friendly substrate that I can migrate our data/stateful workloads to with a view to reducing operational overhead and meeting app devs needs regarding stateful workloads. I want to be able to do this agnostic of my platform choice and would like to treat my data as a first class citizen, sizing workloads for dedicated and shared cluster as necessary.

This is really a 1+N request in that we should:

1. provide a base profile which installs and, if necessary, hardens storageOS/OnDat on a given cluster size. see 'generic install'. To this we could add the ability to upgrade/license the platform. Note that this profile may provide the underlying configuration (could be though of as a stateful 'module' within the concept of 'fleets') for a higher level use case. The use cases prescribe affinity and designate presenters and consumers, which we could abstract
2. explore the various 'use cases' . the installation of prometheus across stateful nodes is there with metrics OOTB, we should probably include that as a sane default. After this the profiles which could build on top can be broadly broken into two areas:

Platform Workload (operational datastores):
ElasticSearch (Logging Backend): docs suggest sane defaults/tuning for ES. E + LK for complete logging backend
InfluxDB (Alternative Logging Backend ): Fairly standard deployment for TS data for logging with Telegraf.
Velero: A common backup solution we see in customer engagements. Something we can break into the requisite components but also a backend solution which could be simple to deploy and maintain.
Prometheus: we already endorse Prom-kube-stack and could plug in with storage OS as a platform.

Customer Workload (application datastores):
Postgres
Redis
Kafka
Cassandra

These need little explanation but in the context of stateful data and the emerging ambition of OnDat to support each one of these (should customers want to license the platform). These are building blocks they will have derived from customers who are running production workload today and wish for support and automation to do so. they may also include helpful points on what StorageOS prefers as defaults for a given workload.

The gap between the use-case material and actual operator driven helm charts is apparent but the path to running supported statefulsets with a view to partnership is clearer. I'd prefer to use best of breed operators and prioritise the platform workloads (Which I view as running in a dedicated cluster and on discreet nodes). Elasticsearch and influxDb + Graf would set up platform backends and give folks a head start on what they need before an leaf cluster or additional workloads are deployed.

I appreciate there is a lot to unpack, looking forward to what folks think :)

[fire-ant]

I left out adding an external Etcd profile but there is precedence in doing so with EKSA for multi-cluster meshing (taking advantage of cluster endpoint awareness and the application off cilium policy across all endpoints in a given cluster).

The thinking there suggest that there may a more distinct underlying topologies:

• multi-cluster with a consistent CNI/CSI needs (single purpose platform)
• disparate cluster with inconsistent CNI/CSI needs (multi purpose platform)

some combination of performance/reliability/security may require that Etcd is a separate entity but the reasoning would need to be convincing as external Etcd is overhead and precludes it from CAPI lifecycle mgmt (needing to be fed and watered discreetly or tied back to CNI upgrades/ownership)

a common component that others may have more customer experience around.

[fire-ant]

I took a trip around the communities and it seems like there is consensus around moving from ES to Loki and usually a toss up for long term metrics storage between thanks/cortex.

I'm curious what the consensus is o. Cortex given it is a Weaveworks product and appears to have at last one other company productising it - https://opstrace.com/blog/introducing-a-cortex-operator. Another one with an operator but which fits the AWS SSP nicely.