Migrate from Promtail to Grafana Alloy

[dominik003]

We currently use Promtail to collect logs from our pods and apply proper labels, but as noted here Promtail is feature complete and new features will be added to Grafana Alloy. Therefore, we should migrate from Promtail to Alloy.

Before migrating, we should evaluate how we can integrate Alloy into the CCM. We are currently using Promtail sidecar containers, which means that each pod where we collect logs consists of two containers, one for the application itself and one for Promtail. Both containers then share a volume where the applications write the logs to and Promtail collects the logs from. We had to choose this method because the simpler method, where Promtail automatically detects pods and extracts logs, was not applicable in our restricted environment (as it required administrator privileges). So in the best case, we can use another simpler method with Alloy.