feat: add opt_out_useragent_filter and $browser_type

[christianvuerings]

Description

We at Cambly would love to capture bot traffic through PostHog. Currently bot traffic is automatically filtered out and there is no way to disable the filter.

posthog-js/src/posthog-core.ts

Line 869 in 958fec4

if (userAgent && _isBlockedUA(userAgent, this.config.custom_blocked_useragents)) {

Changes

• Add opt_out_useragent_filter config option to allow clients to opt out of user agent filtering (e.g. bots)
• Add $browser_type to every capture event to be able to differentiate between bots and regular browsers. Based on User Agent Populator

Checklist

• Tests for new code (see advice on the tests we use)
• Accounted for the impact of any changes across different browsers

[pauldambra]

One thought for safety but otherwise this looks good to me

[christianvuerings]

@pauldambra thank you for the quick review! I noticed that the TestCafe runs fail but it seems like it might be because the jobs on my PR don't have access to BROWSERSTACK_USERNAME or BROWSERSTACK_ACCESS_KEY?

https://github.com/PostHog/posthog-js/actions/runs/7422381529/job/20199038129?pr=949

ERROR Error: Authentication failed. Please assign the correct username and access key to the BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY environment variables.
    at Object.default_1 [as default] (/home/runner/work/posthog-js/posthog-js/node_modules/testcafe-browser-provider-browserstack/src/utils/request-api.js:9:15)

[pauldambra]

Yeah, I think we have to accept that we can't run the testcafe tests on a fork. I guess there's no way not to risk leaking the secrets to a malicious fork.

If this change was larger then I'd be tempted to duplicate it myself and run it locally but I think we'd be safe to merge this and then follow-up if a bug fix was needed

[christianvuerings]

Note that we also added the raw user agent as $useragent when available. That will help us to differentiate between different bot traffic.

Ensured the name lines up with the User Agent Populator documentation

[pauldambra]

Note that we also added the raw user agent as $useragent when available. That will help us to differentiate between different bot traffic.

Ah... we already send $raw_user_agent with each event.

So, you shouldn't need to add the $useragent prop.

If we can drop that then I think this PR is good to go.

(sorry to be so picky... everything that ends up in the SDK is supported forever so I've learned to be careful here 😊)

[christianvuerings]

@pauldambra reverted the $useragent change so we should be good to go.

I did notice that none of our events have a $raw_user_agent as a property in the PostHog dashboard - it also doesn't get send from within the JS client. Do you know if that's filtered out somehow? $browser and $browser_version do show up: https://gist.github.com/christianvuerings/f6b2675f8043a07e348baf68990806be

[pauldambra]

Ah "$lib_version": "1.70.2", $raw_user_agent was added after that version. So, if we get you onto this newer version once it is released then we should kill two birds with one stone 👍