Geyser stream of full blocks is lagging/falling behind

[grooviegermanikus]

Problem

The full block stream cannot keep up with slot progression on the chain and also not with block meta sent via geyser.

Lag is approx 42 slots in 60 seconds (mainnet)!

Did not observe that lag on testnet.

[grooviegermanikus]

the effect is also dependent on the downstream network link speed

[grooviegermanikus]

found that this configuration changes in the GrpcClient helps:

• buffer_size: set to 64k (default: 1024)
• initial_stream_window_size: set to 4194304 (4mb, default: 2mb)

related - there is a documentation issue with h2/hyper regarding these parameters (issue)

[Juanito87]

Are you able to process the stream quick enough? Do you have any kind of bandwidth limitation downstream?
We have observed in the past that if the compute can't manage the data sent by geyser quickly enough, or the bandwidth is not enough to send all data timely, geyser stream will lag behind.

[grooviegermanikus]

the strange thing is that Data gets soaked into black hole. wonder how grpc service could spot that issue.
do you think that there is back pressure somewhere?

[fanatid]

@grooviegermanikus can you please try version 1.14.1? PRs #313 & #315 should improve performance

[brameka]

Did this ever get resolved? I have the same issue subscribing to blocks with transactions enabled. The stream lags immediately. 42 slots behind in 60 seconds on mainnet. Tested my gRPC client on a large AMD epyc machine close to the node so latency and compute shouldn't be a problem. Anyone have some insight PLEAAASE

[fanatid]

@brameka you can try to change window size to very huge one, maybe it was automatically scaled