Detecting URLs, emails, and IP addresses, for Fin Natural language processor.
The core lexer doesn't treat URLs, emails and IP addresses any differently, so it will separate between the www and google and consider them as separate tokens when presented with the string www.google. This is obviously inaccurate, and will lead to many inaccuracies in the POS tagger and the dependency parser
The solution to this problem is to have an preprocessor function that takes out the URLs and a postprocessor function that puts them back after the lexer, the POS tagger and the dependency parser are done with the sentence.
And while we're at it, we can attach a detector to the prototype that gives you the URLs that have been detected.
npm i --save fin-urls
import * as Fin from "finnlp";
import "fin-urls";
const input = "Here's an email [email protected]. and a website: www.google.com."
const instance = new Fin.Run(input);
const result = instance.links();
const links = result.filter((link)=>link);
console.log(result);
console.log(links);The above example will give you:
[
[false,false,false,false,{"type":"email","token":"[email protected]"},false],
[false,false,false,false,{"type":"url","token":"www.google.com"}]
][
[{"type":"email","token":"[email protected]"}],
[{"type":"url","token":"www.google.com"}]
]