Can not send data to pulsar through Pulsar Rest API #14390

ngocchien · 2022-09-13T10:52:22Z

ngocchien
Sep 13, 2022

Hi all, I have a problem about send the messages to Pulsar.

My systems require TLS, so I can not use vector sink type pulsar (because the type pulsar does not support TLS ), then I used sink type HTTP to send data to Pulsar through Pulsar Rest API.
With Pulsar Rest API, the CURL command sends messages to pulsar success.

But when I use vector then It happen an error.

This is my vector configuration :

[sources.dummy_logs]
type = "generator" # required
lines = ["{\"messages\":[{\"key\":\"my-key\",\"payload\":\"data at here\",\"eventTime\":1603045262772,\"sequenceId\":2}]}"]
sequence = false # optional, default, relevant when [`format`](#format)
batch_interval = 5
format = "shuffle"

#sink http
[sinks.http]
type = "http" # required
inputs = [ "dummy_logs" ]
request.headers.Content-Type = "application/json"
request.headers.Accept = "*/*"
request.headers.User-Agent = "PostmanRuntime/7.29.2"
request.headers.Cache-Control = "no-cache"
request.headers.Accept-Encoding = "gzip, deflate, br"
encoding.codec = "text"
uri = "http://192.168.197.135:8080/topics/persistent/common/stream/topic-test-data"


#[sinks.console]
#type = "console"
#inputs = ["dummy_logs"]
#encoding.codec = "text"

This is the error message.

2022-09-13T10:35:06.398321Z ERROR sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 400 Bad Request"
2022-09-13T10:35:06.398458Z ERROR sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::sinks::util::sink: Response failed. response=Response { status: 400, version: HTTP/1.1, headers: {"broker-address": "localhost", "cache-control": "must-revalidate,no-cache,no-store", "content-type": "text/html;charset=iso-8859-1", "content-length": "550", "server": "Jetty(9.4.44.v20210927)"}, body: b"<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=ISO-8859-1\"/>\n<title>Error 400 Bad Request</title>\n</head>\n<body><h2>HTTP ERROR 400 Bad Request</h2>\n<table>\n<tr><th>URI:</th><td>/topics/persistent/common/stream/topic-test-data</td></tr>\n<tr><th>STATUS:</th><td>400</td></tr>\n<tr><th>MESSAGE:</th><td>Bad Request</td></tr>\n<tr><th>SERVLET:</th><td>org.glassfish.jersey.servlet.ServletContainer-1227862a</td></tr>\n</table>\n<hr/><a href=\"https://eclipse.org/jetty\">Powered by Jetty:// 9.4.44.v20210927</a><hr/>\n\n</body>\n</html>\n" }

I have build a Rest API, then I received data from the vector. (I change only URI in configuration-vector.toml to my REST API URL)

This is raw data from my Rest API :

request.data ====>  b'{"messages":[{"key":"my-key","payload":"data at here","eventTime":1603045262772,"sequenceId":2}]}\n'

request.headers ====>  Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache
User-Agent: PostmanRuntime/7.29.2
Host: 192.168.197.135:5000
Content-Length: 98

request.method ====>  POST
request.json ====>  {'messages': [{'key': 'my-key', 'payload': 'data at here', 'eventTime': 1603045262772, 'sequenceId': 2}]}

I don't understand this problem.

Please help me.
Thanks all.
Chien Nguyen.

neuronull · 2022-10-26T19:17:36Z

neuronull
Oct 26, 2022

Hi @ngocchien ,

Apologies for the delayed response!

Firstly, is this still an issue you are facing?

Also, I noticed you mentioned needing TLS and hence using the http sink.
However it looks like no TLS settings were set in your vector configuration for that sink?

HTTP 400 indicates the server thinks there is something wrong with the request 🤔

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can not send data to pulsar through Pulsar Rest API #14390

{{title}}

Replies: 1 comment

{{title}}

Select a reply

Can not send data to pulsar through Pulsar Rest API #14390

ngocchien Sep 13, 2022

Replies: 1 comment

neuronull Oct 26, 2022

ngocchien
Sep 13, 2022

neuronull
Oct 26, 2022