Running Librechat bebind GCP Loadbalancer - Not able to disable IP based login ratelimit

[xueshanf]

What happened?

We are running Librechat behind Google LB.

Our users logins can be easily blocked if there are many users try to login at the same time because the request comes from same IP. They get "message":"Too many login attempts, please try again after 5 minutes."}. The affected users do not have to try many times- sometimes they got the message the first time to try to login.

The only IP related env we can find is LIMIT_MESSAGE_IP=, but it doesn't help login attempts ratelimit based on IP.

Is there a way to workaround this or hidden configuration)s) can tune to disable IP based login ratelimits?

Steps to Reproduce

Configuration:

LOGIN_VIOLATION_SCORE=1
LOGIN_MAX=7
LOGIN_WINDOW=5
ALLOW_EMAIL_LOGIN=false
ALLOW_SOCIAL_LOGIN=true

Have a few users to login/logout at the same time. They would bump into "Too many login attempts, please try again after 5 minutes."}, within one or two tries.

Basically the requests are coming from the same LB IP and shared the same limits.

I would think the behavior would be the same for any reverse proxy, include nginx, no?

short of disable IP based login attempts limits, or use real client IP for ratelimit, would set LOGIN_MAX to a very large number help?

Any help is much appreciated!

Xueshan

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

If the LibreChat API is sitting behind a reverse-proxy/load-balancer, we can still receive the origin IP if it's properly forwarded.

app.set('trust proxy', 1);

You can also experiment with this for testing:

app.set('trust proxy', true);

This should in theory help express.js identify the req.ip, though we can investigate further in Express and load balancer context:

Worth checking:

• That your proxy/load balancer is correctly forwarding the client IP.
• The 'trust proxy' setting in your Express.js configuration.
• The contents of headers like X-Forwarded-For in your incoming requests, on LibreChat side.

simple log statements within the login limiter handler, in api/server/middleware/limiters/loginLimiter.js should help determine this:

console.log('req.ip:', req.ip);
console.log('X-Forwarded-For:', req.headers['x-forwarded-for']);
console.log('X-Real-IP:', req.headers['x-real-ip']);

If all else fails, and if disabling the loginLimiter is an option for you, commenting its usage is easy enough but not recommended.

---

I have my own instance running behind Traefik, and I can confirm from the Login info logs that my IP is recognized as expected (also tested for difference with vpn)

# ips redacted
{"level":"info","message":"[Login] [Login successful] [Username: [email protected]] [Request-IP: x.x.x.43]","timestamp":"2024-09-04T05:04:13.354Z"}
{"level":"info","message":"[Login] [Login successful] [Username: [email protected]] [Request-IP: x.x.x.72]","timestamp":"2024-09-04T05:04:57.846Z"}

[xueshanf]

I did see app.set('trust proxy', 1); in librechat code, and GCP LB also have X-Forwarded-For. Not sure why it’s not taken.

I am trying X-Real-IP now. Will see that take or not and will output request header info for further debugging.

Thank you so much for your helpful and fast reply! I will share our results.

Xueshan

[danny-avila]

You’re welcome! I’m happy to make changes to the codebase to accommodate this but I’m not sure what will work for you.

I didn’t think to use google before but I found this stackoverflow post that might be helpful:
https://stackoverflow.com/questions/37493134/how-to-get-remote-clients-ip-address-in-an-express-based-node-js-app-running-on

[danny-avila]

Perhaps I can add a setting to use a specific req.headers key/value?

[xueshanf]

@danny-avila

I have tested with GCP LB. app.set('trust proxy', true); works, instead of app.set('trust proxy', 1);

I am also able to output logs as you suggested. When app.set('trust proxy', 1), the code doesn't not use client ip (req.ip is LB's address), even X-Forwarded-For and X-Real-IP are both presented in the log. app.set('trust proxy', true); will show req.ip is the real client ip.

I think it might worthwhile to 1. Make an env so to allow customize app.set call. 2. output request ips really helped to nail this down, so a flag to turn it on/off would also be good.

My problem is resolved by app.set('trust proxy', true); Thank you!

[lindner-tj]

Hi Xueshan,

How exactly do you turn on the app.set('trust proxy', true) setting? I'm having the same issue as you, i.e. "message":"Too many login attempts, please try again after 5 minutes."}

kind regards,
Tom

[xueshanf]

We ended up to modify the code api/server/index.js to do app.set(‘trust proxy’, 2), instead of “true” which is to trust all x-forwarded-for. 2 is real client ip when librechat kubernetes pod is directly behind the GCP LB. We don’t run nginx. We turned on debug as Danny suggested to find out the real ip. Since the code now is not configurable, you either create a configmap (if using K8s) to override the index.js in the official release, or build your own image with the modifications. Anther workaround is to set LOGIN_MAX reasonably large to avoid the issue.