GCP PubSub source unable to read messages >4MB

[sbalmos]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

The GCP PubSub source is unable to receive messages >4MB in size, logging an OutOfRange error - "Error, message length too large: found xxx bytes, the limit is: 4193404 bytes". This is because of the Prost >v0.9 breaking fix from a few years ago that changed the default max decode size limit to 4MB. The Vector source/sink and other gRPC / protobuf-based sources/sinks also had this same issue.

The fix would be to call max_decoding_message_size(usize::MAX) as was done in #18186 to the Vector source. Alternatively, the size could be set to a little over 10MB, as GCP PubSub's max supported per-message size is 10MB.

Configuration

No response

Version

0.39.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

#18186