Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(gcp_pubsub source): Remove Tonic default 4MB decode size limit #22091

Merged
merged 5 commits into from
Jan 3, 2025
Merged

fix(gcp_pubsub source): Remove Tonic default 4MB decode size limit #22091

merged 5 commits into from
Jan 3, 2025

Conversation

sbalmos
Copy link
Contributor

@sbalmos sbalmos commented Dec 29, 2024

Summary

Removes the default 4MB message size decoding limit created by Tonic 0.9, preventing the gcp_pubsub source from receiving messages larger than this size.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

How did you test this PR?

Created a test PubSub topic in a personal GCP account, published a >4MB size message to the topic, and read from the subscription successfully.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

  • Please read our Vector contributor resources.
  • [-] If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

@sbalmos sbalmos requested a review from a team as a code owner December 29, 2024 02:19
@github-actions github-actions bot added the domain: sources Anything related to the Vector's sources label Dec 29, 2024
jszwedko
jszwedko approved these changes Jan 2, 2025
View reviewed changes
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix @sbalmos ! Agreed that we should match the old tonic behavior as we did for the vector sink. In the future, we could make this user-configurable.

@jszwedko jszwedko enabled auto-merge January 2, 2025 16:45
@sbalmos
Copy link
Contributor Author

sbalmos commented Jan 2, 2025

This seems to be blocked from merging because the unrelated parse_klog VRL function tests are failing

@jszwedko
Copy link
Member

jszwedko commented Jan 2, 2025

This seems to be blocked from merging because the unrelated parse_klog VRL function tests are failing

Those should be fixed on master now. I'll merge it in here.

@jszwedko jszwedko added this pull request to the merge queue Jan 3, 2025
Merged via the queue into vectordotdev:master with commit 3b90e26 Jan 3, 2025
40 checks passed
@sbalmos sbalmos deleted the pubsub-msg-size branch January 3, 2025 19:34
titaneric pushed a commit to titaneric/vector that referenced this pull request Jan 15, 2025
@sbalmos @jszwedko @titaneric
fix(gcp_pubsub source): Remove Tonic default 4MB decode size limit (v…
f0987bd 
…ectordotdev#22091)

* Remove Tonic default 4MB decode size limit

* Changelog entry

* Update changelog.d/22090_remove_gcp_pubsub_msg_size_limit.fix.md

---------

Co-authored-by: Jesse Szwedko <[email protected]>
Co-authored-by: Jesse Szwedko <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jszwedko jszwedko jszwedko approved these changes

Assignees
No one assigned
Labels
domain: sources Anything related to the Vector's sources
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

GCP PubSub source unable to read messages >4MB
2 participants
@sbalmos @jszwedko