magic number mismatch when upgrading docker to 2.34 -> 2.41

[MostFrabjous]

Description

I attempted to upgrade our percona/pmm-server docker installation from tag 2.34 to 2.41

Expected Results

Upgrade finishes and PMM is accepting data from pmm-agent

Actual Results

PMM does not process new data from pmm-agent (v2.39.0) and logs errors (see below).

Note:

• I could not find any compatibility documentation on pmm-agent version vs pmm-server versions.
• I did not notice anything about pmm-agent when following the Upgrade Documentation.

Version

PMM Server 2.34.0

PMM Agent 2.39

Steps to reproduce

Upgrade docker container: https://docs.percona.com/percona-monitoring-and-management/setting-up/server/docker.html#upgrade

Relevant logs

2024-04-17T12:14:15.157Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/app/vminsert/main.go:167	remoteAddr: "127.0.0.1:47654"; requestURI: /prometheus/api/v1/write; cannot decompress zstd-encoded request with length 340: invalid input: magic number mismatch

### Code of Conduct

- [X] I agree to follow Percona Community Code of Conduct

[BupycHuk]

Hi, are the logs from PMM Client or PMM Server? Do you see these errors after upgrading to 2.41.0 or with 2.34.0?
Our compatability policy for PMM currently is supporting at least 5 minor releases of PMM Client.
I've found similar issue with VictoriaMetrics, not sure if it will help you VictoriaMetrics/VictoriaMetrics#76 (comment)

[MostFrabjous]

After searching again, I found this VictriaMetrics Issue: VictoriaMetrics/VictoriaMetrics#5301

[MostFrabjous]

Hi, are the logs from PMM Client or PMM Server?

These are server logs

Do you see these errors after upgrading to 2.41.0 or with 2.34.0?

After upgrading to 2.41.0 finished and PMM server starts up again. Errors are visible as soon as pmm-agents are sending data.