Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add TLS support to the Pulsar Sink #10888

Open
bemoore34 opened this issue Jan 18, 2022 · 2 comments · May be fixed by #22148
Open

Add TLS support to the Pulsar Sink #10888

bemoore34 opened this issue Jan 18, 2022 · 2 comments · May be fixed by #22148
Labels
sink: pulsar Anything `pulsar` sink related type: enhancement A value-adding code change that enhances its existing functionality.

Comments

@bemoore34
Copy link

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Current Vector Version

vector 0.19.0

Use-cases

Apache Pulsar supports enabling TLS encryption on client connections to the broker to secure data in transit. Many production instances will have TLS encryption enabled on their Pulsar clusters. It would be beneficial for the Vector Pulsar sink to support TLS connections to Pulsar topics.

A potential workaround is to use the Websocket API on Puslar and send messages to topics using the Vector HTTP sink, which supports TLS. This may not be an option for all users running TLS-enabled Puslar clusters and it would be useful to support TLS with native Pulsar client connections.

Attempted Solutions

Tested the existing sink to a TLS-encrypted cluster and confirmed on discord that it isn't currently supported.

Proposal

Add configuration options for connecting the Vector Pulsar sink to Pulsar topics with TLS 1.2+ enabled.

Reference

https://pulsar.apache.org/docs/en/security-tls-transport/
@spencergilbert spencergilbert added sink: pulsar Anything `pulsar` sink related type: enhancement A value-adding code change that enhances its existing functionality. labels Jan 18, 2022
@zamazan4ik
Copy link
Contributor

@bemoore34 Could you please share the details, how exactly Vector with enabled TLS for Pulsar Sink (it is enabled via specifying Pulsar endpoint with pulsar+ssl prefix) did not work? As far as I see from the current sources, it should work even without additional parameters.

The only potentially useful parameter here is adding custom certificates to the certificate chain. But even without them TLS connection shoould work if your Pulsar certificate is trusted by system trust root.

@bemoore34
Copy link
Author

Sorry, I don't remember the details of what I tested and cannot find my notes. I tried to connect using TLS (pulsar+ssl) and couldn't get it to work. I'm not sure what validation I performed around the certificate trust. I have been using internal certificates and would prefer to have the option to provide a custom CA cert so the Pulsar client trusts the signed cert. If I find my notes I'll confirm the details. I gave up trying after confirming it wasn't supported in Discord.

@pomacanthidae pomacanthidae linked a pull request Jan 9, 2025 that will close this issue
Open
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sink: pulsar Anything `pulsar` sink related type: enhancement A value-adding code change that enhances its existing functionality.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(pulsar sink): support tls options pomacanthidae/vector
3 participants
@bemoore34 @zamazan4ik @spencergilbert