Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IPIntel API support. #33339

Open
wants to merge 30 commits into
base: master
Choose a base branch
from

Conversation

VasilisThePikachu
Copy link
Member

@VasilisThePikachu VasilisThePikachu commented Nov 15, 2024

About the PR

This PR adds support for the IPIntel API and adds it in the connection manager.

Why / Balance

  1. Small servers may not have the time to add a bunch of VPN ranges
  2. Even for big servers, adding a bunch of ranges is tasking and requires manual intervention.
  3. Those IP ranges may at any day become outdated and lead to false positives.
  4. Ban evaders frequently use VPN's to bypass bans.

Technical details

Welcome to what I like to call "return spaghetti"

BREATH IN
This function is at the very end of the connection manager and should STAY THERE AT ALL COSTS. Since we have an API limit to abide by, Anyone working with this codebase should prioritize trying to not make a request to IPIntel if possible. To enable this system, you need to manually enable game.ipintel_enabled in your config. Along with a contact email for the IPIntel service game.ipintel_contact_email

IF YOU DON'T PROVIDE THESE TWO CCVARS IT WILL NOT FUNCTION

If the player has Datacenter or BlacklistedRange ban exemptions, then they get to bypass this system. It's not by any means perfect, so administrators should be expected to receive false positives and act accordingly for their player base. Ensure your admins are aware of the ban_exemption_update and ban_exemption_get commands.

Next up, the player's total playtime is checked. By default, game.ipintel_exempt_playtime is set to 5 Hours. This is in an effort to preserve the API limit for big servers. You can disable this by setting it to 0 if you really hate this.

Next, we will check our cache. Every time we make a request to IPIntel we cache the response for 7 days (game.ipintel_cache_length) since we want to hit IPIntel as little as possible. Again for our API limit. If we got one that's still valid, we use it. If we already have an old record, we mark it as expired and move forward. The same if we are seeing it for the first time.

Ratelimits are terrible and I hate them

A quick check for our Email happens before continuing to preparing the API request. game.ipintel_baseurl controls the base API URL, but you probably don't want to change it unless you decide to go request for a paid plan. We will use the b flag by default as we don't care about if the IP of the player is compromised. (You can change the flag with the game.ipintel_flags ccvar) Just that if it's a datacenter. It also reduces the request time (assuming no latency) to 120ms~ from 200ms-5seconds for a regular scan.

Finally comes the decision. We will now compare what we got from the API. If what we got is bigger, then game.ipintel_bad_rating then we deny the connection if game.ipintel_reject_badwith an optional reject notice to the admin (game.ipintel_alert_admin_rejected). Optionally, if you set game.ipintel_alert_admin_warn_rating to something, and we did not just reject the connection, then you can warn in game admins.

If we had any errors during this process, we will return an unknown result. Whether we kick or not is determined with if game.ipintel_reject_unknown is true or not

Media

game.ipintel_alert_admin_rejected enabled
image

game.ipintel_alert_admin_warn_rating enabled
image

Default Disconnect message to blocked players
image

Requirements

Changelog

🆑 Myra
ADMIN:

  • add: Added IPIntel's API to the connection code. If enabled, this will automate VPN/Proxy detection.

@github-actions github-actions bot added size/L Denotes a PR that changes 1000-4999 lines. S: Untriaged Status: Indicates an item has not been triaged and doesn't have appropriate labels. labels Nov 15, 2024
@VasilisThePikachu VasilisThePikachu added A: Admin Tooling Area: Admin tooling and moderation. T: Of Admin Interest Type: Affects administration work a lot, and might require admins to weigh in on and removed S: Untriaged Status: Indicates an item has not been triaged and doesn't have appropriate labels. labels Nov 15, 2024
Copy link
Member Author

@VasilisThePikachu VasilisThePikachu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self minor nitpicks

Resources/Locale/en-US/connection-messages.ftl Outdated Show resolved Hide resolved
@SaphireLattice SaphireLattice added P1: High Priority: Higher priority than other items, but isn't an emergency. D0: Very High Difficulty: Codebase knowledge required is extreme. Consider atomizing instead. S: Needs Review Status: Requires additional reviews before being fully accepted S: Approved Status: Reviewed and approved by at least one maintainer; a PR may require another approval. labels Nov 15, 2024
@github-actions github-actions bot added size/XL Denotes a PR that changes 5000+ lines. and removed size/L Denotes a PR that changes 1000-4999 lines. labels Nov 18, 2024
@VasilisThePikachu VasilisThePikachu marked this pull request as ready for review November 18, 2024 20:05
Content.Server.Database/Model.cs Outdated Show resolved Hide resolved
Content.Server.Database/Model.cs Show resolved Hide resolved
Content.Server.Database/Model.cs Show resolved Hide resolved
Content.Shared/CCVar/CCVars.Game.cs Show resolved Hide resolved
Content.Shared/CCVar/CCVars.Game.cs Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Database/ServerDbManager.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
@github-actions github-actions bot removed size/L Denotes a PR that changes 1000-4999 lines. S: Merge Conflict Status: Needs to resolve merge conflicts before it can be accepted labels Nov 22, 2024
@VasilisThePikachu VasilisThePikachu requested review from PJB3005 and removed request for PJB3005 November 22, 2024 17:46
Content.Server.Database/Model.cs Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Copy link
Member

@PJB3005 PJB3005 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to demand you pull this out into a helper class so that this logic can be thoroughly unit tested. Right now I'm seeing many bugs just through simple review, so I do not trust you've done extensive enough testing manually for me to be confident to deploy this on the servers directly.

Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server.Database/Model.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Content.Server/Connection/ConnectionManager.IPIntel.cs Outdated Show resolved Hide resolved
Not yet in a helper class just wanna do these and commiting before starting that
Copy link
Contributor

@FairlySadPanda FairlySadPanda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have only gone through the ConnectionManager.IpIntel.cs file in anger;

This can be expressed quite a bit more clearly, I'll talk to you directly about the strategy I took when doing a tidy-up.

The rate-limiting code is superfluous. Failing due to a rate limit can be handled as a backoff; it's up to the IPIntel API to decide when we're un-rate-limited, not us, so we can't really programatically decide we're not rate-limited anymore.

Removing this makes us now really only have to worry about doing unit testing that can mock the request. I've taken a look and unfortunately it doesn't look like there's a mock imlementation of IHttpClientHolder, so you'll need to dig into the Mock<> route to make the unit tests.

To give you an idea where I got to with tidying the code:

private async Task<(bool IsBad, string Reason)> ProcessResponse(NetConnectingArgs e, HttpResponseMessage request)
    {
        switch (request.StatusCode)
        {
            case HttpStatusCode.OK:
                var score = Parse.Float(await request.Content.ReadAsStringAsync());

                if (!await _db.UpsertIPIntelCache(DateTime.UtcNow, e.IP.Address, score))
                {
                    _sawmill.Error(
                        $"Unable to upsert an IP address '{e.IP.Address}' into the IPIntel cache on the database.");
                }

                return IsScoreTooHigh(score, e);
            case HttpStatusCode.TooManyRequests:
                _sawmill.Warning($"We've hit the IPIntel request limit.");
                _failedRequests++;
                _releasePeriod = TimeSpan.FromSeconds(_failedRequests * _backoffSeconds);

                return _isRejectClientsOnRateLimitReached
                    ? (true, Loc.GetString("ipintel-server-ratelimited"))
                    : FalseResp;
            case HttpStatusCode.BadRequest:
                var errResp = await request.Content.ReadAsStringAsync();

                if (IsKnownError(errResp, out var rejectResult))
                    return rejectResult;

                // Oh boy, we don't know this error.
                _sawmill.Error(
                    $"IPIntel returned {errResp} (Status code: {request.StatusCode})... we don't know what this error code is. Please make an issue in upstream!");

                return _rejectUnknown ? (true, Loc.GetString("ipintel-unknown")) : FalseResp;
            default:
                _sawmill.Warning(
                    $"Unexpected response status code when querying IPIntel: '{request.StatusCode.ToString()}'. Treating as an unknown response.");
                _failedRequests++;
                _releasePeriod = TimeSpan.FromSeconds(_failedRequests * _backoffSeconds);

                return _rejectUnknown ? (true, Loc.GetString("ipintel-unknown")) : FalseResp;
        }
    }

The above demonstrates a simple switch approach to processing HTTP status codes.

I would strongly recommend treating the else keyword as something to avoid unless it's trivial code. Switches are a far clearer way of expressing an if-else chain.

@github-actions github-actions bot added the S: Merge Conflict Status: Needs to resolve merge conflicts before it can be accepted label Dec 16, 2024
Copy link
Contributor

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@github-actions github-actions bot removed the S: Merge Conflict Status: Needs to resolve merge conflicts before it can be accepted label Dec 26, 2024
Copy link
Member

@PJB3005 PJB3005 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ship it.

@PJB3005 PJB3005 added the S: Needs Engine Update Status: Requires a newer version of the Robust Toolbox engine. label Jan 5, 2025
@PJB3005
Copy link
Member

PJB3005 commented Jan 5, 2025

(Don't actually we need to update the engine first)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: Admin Tooling Area: Admin tooling and moderation. D0: Very High Difficulty: Codebase knowledge required is extreme. Consider atomizing instead. P1: High Priority: Higher priority than other items, but isn't an emergency. S: Approved Status: Reviewed and approved by at least one maintainer; a PR may require another approval. S: Needs Engine Update Status: Requires a newer version of the Robust Toolbox engine. S: Needs Review Status: Requires additional reviews before being fully accepted size/XL Denotes a PR that changes 5000+ lines. T: Of Admin Interest Type: Affects administration work a lot, and might require admins to weigh in on
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants