pipeline 🤝 replay ingestion changes

[pauldambra]

Feature request

pipeline and replay teams are going to trade time to improve heatmap_data and $exception event ingestion

## why change heatmap data ingestion?

1. we get many heatmap_data items per event that carries them, so if we're under heavy load we automatically take a multiple of that load and it's hard to scale/react because the magnification is happening inside main event processing
2. we want to make these changes without breaking main event ingestion. Also improves development speed by proxy.
3. failure isolation, e.g. incident time we have more easier leavers we can pull
4. not slowing down analytics ingestion
5. cost we can optimize the independent clearly different work

TODO

• move $heatmap_data from being a passenger on other events to on its own $$heatmap event @pauldambra
  • feat: move heatmaps to their own event posthog-js#1287
• update ingestion runner to make sure heatmap data keeps flowing (if necessary) @pauldambra
  • feat: separate heatmap processing #23505
• create kafka topic for heatmap raw topic - team pipeline
• add a new plugin-server role and deployment for heatmap (running dupe of historical ingestion code, i.e. analytics without overflow) - team pipeline
• update capture-rs to send $$heatmap events are written to dedicated kafka topic (we won't be changing capture-py as it's going to die soon) @xvello
• Update plugin-server code to optimize heatmap role so it only does validation and writing to the heatmap ingestion topic. It can't do any other processing (it can do no other processing no $set, no exports etc ( heatmaps are free so we'll keep processing cheap) - team pipeline
  - team look-up for token resolution & are heatmaps enabled or not
  - no PG (no persons, groups, ...)
  - no processEvent plugins
  - event written to a dedicated table
• investigate writing one message to CH kafka topic which is exploded in the materialized view that ingests them instead of sending one kafka message per heatmap data item @pauldambra ??

why change $exception data ingestion

1. we want to add more processing to these events, that will require changes to speed of processing, infra requirements, etc, we want to make these changes without breaking main event ingestion. Also improves development speed by proxy.
2. failure isolation, e.g. incident time we have more easier leavers we can pull
3. not slowing down analytics ingestion
4. cost we can optimize the independent clearly different work

TODO

• setup /i/v0/x ingestion route
• make sure any $exception event can be configured to be sent to the new route @pauldambra
  • feat: send errors one way posthog-js#1289
• add topic, capture-rs, new plugin-server role & deployment todo's as above - team pipeline
• Update plugin-server code to optimize exception role so it only does things it needs - team pipeline
  - only person lookup (no writes)
  - keep processEvent plugins
  - keep groups resolution and PoE

Debug info

No response

[pauldambra]

@tiina303 dumped my thoughts here since we've probably gone beyond slack

i'd be happy to discover i'm wrong so feel free to say what tasks i'm missing / or can delete / or shouldn't be trying to do etc etc