Add information about other langID models

[Uinelj]

see #106

[Uinelj]

CI is failing because of CC 503 errors :(
We should maybe embed simpler CC shards.

[chris-ha458]

This is almost exactly what I expected. (I forgot all other details)

CI is failing because of CC 503 errors :(
We should maybe embed simpler CC shards.

This means that the build test is pulling CC shards that are unavailable due to HTTP 503 errors(likely from overload) right?
It is a bit unfortunate that issues like this is happening for a "benign documentation change" but I guess this is setup this way for a reason.
Is there anything I could do to help?

[Uinelj]

Downloading shards at CI run might be a smell, since you're dependent on the network for your CI to run. My goal was to integrate real-world data to be sure nothing breaks :)

We could split testing into two different parts: one with tiny custom shards (that we can build by filtering already existing ones, only keeping wikipedia content to avoid copyright issues), and one that can be launched locally with shards from the internet.

I'm interested in keeping the "real world shards" because a lot of noisy/bad data is in it (such as images encoded as text, stuff like that) and that data can break things so it's important to test on.