Integrating spacy-huggingface-pipelines and refactoring NlpEngine logic

[omri374]

Change Description

This PR improves the handling of transformers models using the NlpEngine flow. Note that there are two ways to introduce NER models into presidio: NlpEngine and a standalone recognizer. See more info on #1083.

Main changes:

1. The entire handling of models, entities, mapping and alignment of model spans to Presidio's spans is done as part of the NlpEngine config (either SpacyNlpEngine or TransformersNlpEngine). In the future, we could also add FlairNlpEngine or other NER packages.
2. Introduced a new NerModelConfiguration dataclass which holds the user configuration. Configuration can be set in a conf file, for example:
   
   with most of the configuration coming from here: https://github.com/explosion/spacy-huggingface-pipelines#token-classification
3. Simplified the logic in SpacyRecognizer which now only gets the entities out of NlpArtifacts and returns them
4. Integrated spacy-huggingface-pipelines to improve the handling of transformer models inside spaCy pipelines.

Flow before:

sequenceDiagram
    AnalyzerEngine->>SpacyNlpEngine: Call engine.process_text(text) <br>to get model results
    SpacyNlpEngine->>Spacy: call spaCy NER model
    Spacy->>SpacyNlpEngine: return PII entities
    SpacyNlpEngine->>AnalyzerEngine: Pass NlpArtifacts<BR>(Entities, lemmas, tokens etc.)
    Note over AnalyzerEngine: Call all recognizers
    AnalyzerEngine->>SpacyRecognizer: Pass NlpArtifacts
    Note over SpacyRecognizer: Extract PII entities out of NlpArtifacts, <BR>map to Presidio entities, <BR>filter requested entities, <BR> add confidence scores
    SpacyRecognizer->>AnalyzerEngine: Return List[RecognizerResult]

Loading

Flow after:

sequenceDiagram
    AnalyzerEngine->>SpacyNlpEngine: Call engine.process_text(text) <br>to get model results
    SpacyNlpEngine->>Spacy: call spaCy NER model
    Spacy->>SpacyNlpEngine: return PII entities
    Note over SpacyNlpEngine: Map entity names to Presidio's, <BR>update scores, <BR>remove unwanted entities <BR> based on NerModelConfiguration
    SpacyNlpEngine->>AnalyzerEngine: Pass NlpArtifacts<BR>(Entities, lemmas, tokens, scores etc.)
    Note over AnalyzerEngine: Call all recognizers
    AnalyzerEngine->>SpacyRecognizer: Pass NlpArtifacts
    Note over SpacyRecognizer: Extract PII entities out of NlpArtifacts
    SpacyRecognizer->>AnalyzerEngine: Return List[RecognizerResult]

Loading

Note to reviewer: A PR with updates to docs can be found here #1177

Issue reference

This PR fixes issue #1083

Checklist

• I have reviewed the contribution guidelines
• I have signed the CLA (if required)
• My code includes unit tests: partially/wip
• All unit tests and lint checks pass locally
• My PR contains documentation updates / additions if required: wip

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[LSD-98]

Hello @omri374,

By running some test i got a warning from spacy-huggingface-pipeline:
UserWarning: Skipping annotation, {'entity_group': 'LOC', 'score': 0.46050456, 'word': 'ian', 'start': 33, 'end': 36} is overlapping or can't be aligned for doc 'Je suis Tom et je travaille à Ardian'

In the end, it detected the right entities to anonymize but I wondered whether it could lead to a "false negative" (i.e. skipping an entity that should be detected). I saw you also had the same issue here. I did not understand the technical points of the answer so I wanted to check if you got a solution for this.

Many thanks in advance!

PS : please tell me if I should post here or on the discussion tab.

[omri374]

@LSD-98 thanks for raising this. This is caused by a mismatch between wordpiece tokenization (in transformers) and the spaCy tokenizer. It is defined as "expand" as usually wordpiece tokens are a subset of a spaCy token, but there could be cases where this might result in a false negative or false positive (most likely false positive, as the wordpiece token would be expanded to more than the PII itself). I don't see an immediate workaround for this, so I guess we'd have to live with it. If someone is making sure there are no alignment errors, they could still add a transformers recognizer as an independent recognizer and not as part of the NLPEngine mechanism.

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[niwilso]

Overall looks great! Thank you for providing the swimlane flow diagram.

Just a few points to address before approval.