Is there a performance issue with the http module ?

[PanHywel]

When I use http as input source and distribute it through Nginx in front, I find that there are a lot of error logs in Nginx. I don't know if this is limited by the performance of the http module. But I didn't find any relevant information about the configuration in the configuration documentation.

The flow is as follows.
Nginx upstream => vector http => transforms => kafka

Nginx error message.

connect() failed (111: Connection refused) while connecting to upstream

Addendum: I have a very high volume of Nginx requests, and not all of them are rejected, but most of them are.

[jszwedko]

Hmm, that's interesting. Would you mind sharing the Nginx and Vector configurations (also version numbers)? I can't think of anything offhand that would cause that.

[PanHywel]

@jszwedko Hi, this is my configuration

nginx version:4.8.5;
nginx conf

upstream upsvr{
    server 172.16.1.41:7001 weight=1;
    server 172.16.1.41:7002 weight=1;
    server 172.16.1.41:7003 weight=1;
}

server {
        listen :80;

        location / {
                client_body_buffer_size 30m;
                client_max_body_size    50m;
                proxy_set_header X-Real-IP        $remote_addr;
                proxy_set_header X-Forwarded-For  $remote_addr;
                proxy_set_header Host $host;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_pass http://upsvr;
        }

}

vector version: vector 0.15.0 (x86_64-unknown-linux-gnu 994d812 2021-07-16)

[sources.httpSources]
type = "http"
address = "172.16.1.41:7001"
path = "/client-lua"
encoding = "json"
headers = ["x-forwarded-for","User-Agent"]

[transforms.parse_logs]
type = "remap"
inputs = ["httpSources"]
source = '''
. = unnest!(.msg)
#.clientIp = del(."x-forwarded-for")
'''

[transforms.parse_logs_two]
type = "remap"
inputs = ["parse_logs"]
source = '''
.appName = .msg.appName
.confId = .msg.confId
.text = .msg.text
.time = .msg.time
.type = .msg.type
.userId = .msg.userId
.clientIp = del(."x-forwarded-for")
.userAgent =del(."User-Agent")
.siteDomain = msg.siteDomain
.userName = msg.userName
.playerVersion = msg.playerVersion
.playerMode = msg.playerMode
.siteId = msg.siteId
#.localTime = del(.timestamp)
.localTime = format_timestamp!(now(),format:"%F %T")
#del(.timestamp)
del(.source_type)
del(.path)
del(.msg)
'''



[sinks.sink_kafka]
 type = "kafka"
 inputs = ["parse_logs_two"]
 bootstrap_servers = "172.16.1.41:9092"
 # key_field = "user_id"
 topic = "http-vector-kafka"

 encoding.codec = "json"
 # OPTIONAL - Buffer
 [sinks.sink_kafka.buffer]
  type = "memory" # default, enum
  max_events = 500
  max_size = 104900000
  when_full = "block" # default, enum

[jszwedko]

Thanks for the config @PanHywel ! Would you be able to share more details about the throughput you are trying to achieve? Requests / second and size of each request?

I attempted to reproduce, but was unable to trigger this using locust to load test at ~60 requests / second. I wasn't using Kafka though, I was just sending to a blackhole sink. I'm curious if you see the same behavior if you try using the blackhole sink instead of Kafka? If not, that could point to there being back-pressure from the kafka sink causing requests to build up in the source. Eventually you might hit a limit of open file descriptions. You could verify that by looking at the number of open connections to Vector when you observe those error messages.

[PanHywel]

@jszwedko I apologize for only getting back to you now, my request count is ~100000/sec, this is an estimate and it may actually be more than that. I will use blackhole sink for testing.

[JeanMertz]

@PanHywel did you ever get around to testing this using the blackhole sink? Also, we've had a lot of performance releases in the last year, so hopefully your issue has since been resolved in one of those releases.

I'll mark the issue as tentatively resolved, but can unresolve it if you are still experiencing problems.