add domain parsing with custom psl list and removing false positives

[mindofmar]

This PR introduces a major improvement to the test-lists.ts file which is critical for preventing false positives within CICD.

Rationale

Previously, we've been allowing subdomains to bypass the tranco check because we were not parsing the domain within this test. Therefore auth.magic.link !== magic.link even though magic.link is on the tranco list and this subdomain is owned by the same company. This has been the source of widespread impact false positives being merged into this repo, and this PR will reduce future cases of the problem. (with the assumption that the tranco list stays up to date so that it contains future popular websites).

As you can see from the websites I removed in config.json including berkeley.edu, bitcoin.com, consensys.net and others, this does a very good job at detecting false positives correctly.

Extending public suffix list

The change that enables this change is the custom-tlds.ts file which now extends public suffix list with suspected hosting providers that do NOT exist on PSL already. This may be annoying to keep up to date, but is necessary to do because PSL is slow to update and there are lots of hosting providers that allow you to host malicious websites. Unfortunately, these hosting providers have a high enough tranco score so they may cause the CICD to fail.

To remedy this you have 2 options:

1. If only 1 website in config.json uses this domain, you can just add the hostname to the bypass list inside test-lists.ts
2. If several websites are using this as a hosting provider OR it the root website is advertising DNS services, add it to the custom-tlds.ts file.

To debug the above, I recommend the following code snippet (or CTRL+F 😉 )

// test-lists.ts

            import { parse } from 'tldts';
           //........

            t.equal(blocked.length, 0, `The following domains should not be blocked: ${blocked}`);

            const map = new Map();
            for (let x = 0; x < blocked.length; x++) {
                const parsedDomain = parse(blocked[x], { allowPrivateDomains: true }).domain;
                if (map.has(parsedDomain)) {
                    map.set(parsedDomain, map.get(parsedDomain) + 1);
                } else {
                    map.set(parsedDomain, 1);
                }
            }
            const sortedMap = new Map([...map.entries()].sort((a, b) => b[1] - a[1]));
            sortedMap.forEach((val, key) => {
                console.log(`${key} : ${val}`);
            });

            t.end();

[socket-security]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@metamask/[email protected]	None	0	344 kB	metamaskbot
npm/@metamask/[email protected]	network	0	259 kB	danfinlay, gudahtt, kumavis, ...8 more
npm/@metamask/[email protected]	network	0	305 kB	danfinlay, gudahtt, kumavis, ...8 more
npm/@scure/[email protected]	None	0	133 kB	paulmillr
npm/@types/[email protected]	None	0	13.9 kB	types
npm/[email protected]	None	0	31.4 kB	ljharb
npm/[email protected]	None	0	212 kB	remusao
npm/[email protected]	None	0	2.97 MB	remusao

🚮 Removed packages: npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@esbuild/[email protected], npm/@metamask/[email protected], npm/@metamask/[email protected], npm/@metamask/[email protected], npm/@scure/[email protected], npm/@types/[email protected], npm/[email protected]

View full report↗︎

[socket-security]

🚨 Potential security issues detected. Learn more about Socket for GitHub ↗︎

To accept the risk, merge this PR and you will not be notified again.

Alert	Package	Note	Source	CI
Network access	npm/@metamask/[email protected]	Module: globalThis["fetch"] Location: Package overview	package.json yarn.lock	🚫
Network access	npm/@metamask/[email protected]	Module: globalThis["fetch"] Location: Package overview	yarn.lock	🚫

View full report↗︎

Next steps

What is network access?

This module accesses the network.

Packages should remove all network access that is functionally unnecessary. Consumers should audit network access to ensure legitimate use.

Take a deeper look at the dependency

Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support [AT] socket [DOT] dev.

Remove the package

If you happen to install a dependency that Socket reports as Known Malware you should immediately remove it and select a different dependency. For other alert types, you may may wish to investigate alternative packages or consider if there are other ways to mitigate the specific risk posed by the dependency.

Mark a package as acceptable risk

To ignore an alert, reply with a comment starting with @SocketSecurity ignore followed by a space separated list of ecosystem/package-name@version specifiers. e.g. @SocketSecurity ignore npm/[email protected] or ignore all packages with @SocketSecurity ignore-all

• @SocketSecurity ignore npm/@metamask/[email protected]
• @SocketSecurity ignore npm/@metamask/[email protected]

[AugmentedMode]

LGTM

[AugmentedMode]

LGTM thanks for making that change