Document using external ClickHouse cluster with Altinity Cloud

[macobo]

Proposed change

In #276 it became theoretically possible to use an external clickhouse cluster.

In practice though, using the cluster requires quite a few fixes to land in a release of PostHog which will take ~month.

This issue documents these fixes and how to get everything working.

Required PRs

• Make service requirements less strict for clickhouse posthog#8311
• Update clickhouse-orm posthog#8314 / Support altinity cloud when fetching server version infi.clickhouse_orm#7
• Allow overriding kafka host for clickhouse via KAFKA_URL_CLICKHOUSE env var posthog#8332
• Allow disabling protobuf with CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS env posthog#8333

Also relevant: PostHog/posthog#8334

Testing the changes and sample values.yaml

I tested out a build where all the above fixes were baked in. The procedure was:

• ngrok tcp 9092 to expose my clickhouse installation to the world. Note down the exposed url and replace it within the values file below
• helm install -f altinity-cloud-values.yaml --timeout 20m --create-namespace --namespace posthog posthog charts/posthog/ --atomic --wait --wait-for-jobs --debug
• kubectl port-forward service/posthog-posthog-kafka 9092:9092 -n posthog while helm is running to expose kafka

The values file required some annoying overrides:

cloud: "private"

image:
  repository: XXX
  sha: XXX

env:
- name: KAFKA_URL_FOR_CLICKHOUSE
  value: "kafka://4.tcp.ngrok.io:10912"
- name: CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS
  value: "1"

kafka:
  advertisedListeners: "INTERNAL://$(MY_POD_NAME).posthog-posthog-kafka-headless.posthog.svc.cluster.local:9093,CLIENT://4.tcp.ngrok.io:10912"

clickhouse:
  enabled: false

externalClickhouse:
  host: xxx.demo.altinity.cloud
  user: xxx
  password: "xxx"
  cluster: posthog
  secure: true

Notes for the future

We should figure out a more stream-lined way of exposing kafka here - there's a bit of a catch-22 going on with advertisedListeners and migrations - we need to know kafka external hostname by the time of migrations.

Alternatives

As an alternative to the protobuf change, users could also manually upload the protobuf file by:

1. Opening the cluster, clicking Configure > Settings
2. Upload these two settings

Note that this needs to be done prior to first helm install and events.proto is checked into our current codebase.

The rest of linked PRs still remain required though.

[yakkomajuri]

SKIP_SERVICE_VERSION_REQUIREMENTS is no longer required

[macobo]

SKIP_SERVICE_VERSION_REQUIREMENTS is no longer required

Only after release 1.33.0 which hasn't happened yet. :)

[macobo]

Me and @guidoiaquinti tried this yesterday. We didn't finish due to a couple of reasons:

1. This required getting the Kafka IP, hence deploying first and then updating the config. This turned out to be problematic for later
   • Also left behind a clickhouse pod even if the new stack had it disabled
2. Kafka topic contained a poison pill due to initially enabling Protobuf and we don't skip broken messages by default in kafka. State of "Protobuf" format schema for clickhouse posthog#8334 we should remove protobuf entirely
3. We initially tried a clickhouse cluster with a dash in the name which caused issues Keep Kafka tables in sync #317
4. We needed to drop the database as we changed our kafka configuration, see Keep Kafka tables in sync #317 for a proposed solution
5. Debugging clickhouse <-> kafka connection issues was fundamentally tricky as selecting from the topic didn't get great results. Logs sometimes indicated connection errors, other times assignment errors.

values.yaml

cloud: "private"

env:
  - name: KAFKA_URL_FOR_CLICKHOUSE
    value: "kafka://IP:9094"

  - name: CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS
    value: "1"

kafka:
  externalAccess:
    enabled: true
    service:
      type: LoadBalancer
      ports:
        external: 9094
    autoDiscovery:
      enabled: true
  serviceAccount:
    create: true
  rbac:
    create: true

clickhouse:
  enabled: false

externalClickhouse:
  host: x.demo.altinity.cloud
  user: admin
  password: "xxxxxxx"
  cluster: "x"
  secure: true

Additionally, I think there's some clever values.yaml wrangling we could do to make spinning up only one component easier.

Will continue on this and document progress here.

[macobo]

Got it working on my second try.

I initially deployed the following values.yaml to get the kafka IP:

cloud: "do"

env:
  - name: CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS
    value: "1"

kafka:
  enabled: true
  externalAccess:
    enabled: true
    service:
      type: LoadBalancer
      ports:
        external: 9094
    autoDiscovery:
      enabled: true
  serviceAccount:
    create: true
  rbac:
    create: true


clickhouse:
  enabled: false

redis:
  enabled: false

postgresql:
  enabled: false

pgbouncer:
  enabled: false

plugins:
  enabled: false

worker:
  enabled: false

web:
  enabled: false

events:
  enabled: false

migrate:
  enabled: false

Got the IP from kubectl get svc -n posthog | grep kafka-0-external

And then:

cloud: "do"

env:
  - name: KAFKA_URL_FOR_CLICKHOUSE
    value: "kafka://IP:9094"

  - name: CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS
    value: "1"

kafka:
  enabled: true
  externalAccess:
    enabled: true
    service:
      type: LoadBalancer
      ports:
        external: 9094
    autoDiscovery:
      enabled: true
  serviceAccount:
    create: true
  rbac:
    create: true

externalClickhouse:
  host: X.demo.altinity.cloud
  user: admin
  password: "X"
  cluster: "X"
  secure: true

clickhouse:
  enabled: false

Will create documentation as this as a basis.