When the broker stops, the subscriber consumer lag

[yp19910928]

Description

I have a Kafka cluster with 5 brokers and 5 controllers. The 5 brokers are grouped as H: (H1/H2), N: (N1/N2), and D: (D1). I have subscriber programs running on the VMs of H1/H2/N1/N2 brokers, which subscribe to the same topic, and send 1 message per second to the topic. When I stop the N1/N2/D1 broker services, the publisher program can publish messages normally, and the subscriber programs on the H1/H2 VMs are also normal. My expectation is that the subscriber programs on the N1/N2 VMs can also subscribe to messages normally, but in reality, the subscriber programs on the N1/N2 VMs can only consume partial Partition messages, and some Partition messages have consumer lag.

H1:5245 H2:5246 N1:5247 N2:5248 D1:52104

How to reproduce

Checklist

Please provide the following information:

• A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• Confluent.Kafka nuget version.
• Apache Kafka version.
• Client configuration.
• Operating system.
• Provide logs (with "debug" : "..." as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.

[yp19910928]

Kafka Broker Config:
num.partitions=4
offsets.topic.replication.factor=4
transaction.state.log.replication.factor=4
transaction.state.log.min.isr=2
default.replication.factor=4
min.insync.replicas=2

Apache Kafka version:
3.3.1

Operating system.
Linux photon 4.19.132-6.ph3 #1-photon SMP Sun Aug 9 02:22:28 UTC 2020 x86_64 GNU/Linux

Confluent.Kafka nuget version
2.3.0