[backend/frontend] Implement full text search for documents (#1483)

[SouadHadjiat]

Proposed changes

PR with first version of imported files indexing and full text search for enterprise edition

• Creation of files index (opencti_files) and configuration of attachment processor
• Adding FileIndexManager module to index all imported files (global and in entities)
• In global search, adding button to extend the search to indexed files
• handle entities markings / organization restriction changes : update indexed files to have the same data restrictions

First part of file index management panel :

• Adding "File indexing" menu in settings top bar
• Display enterprise edition info if not enabled
• Display requirements for file indexing (elasticsearch / opensearch)
• Display number and size of files that will be indexed
• Manage file indexing : start / pause, display number and size of files indexed

Indexation is paused by default.

About file indexing :
We support these mime types : text/plain, text/csv, application/pdf, application/vnd.ms-excel, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet (excel sheets)

Related issues

• Full text search for documents #1483

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant uses case
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality

Further comments

https://www.notion.so/filigran/Management-panel-for-File-indexing-de5a1879c62f41af99943c58a30f4aa6

https://www.notion.so/filigran/Full-text-search-for-documents-11a67658fc37410cbbbdde2db78da73a#3281f177046749678fdbb8539b5cbaab

[Archidoit]

Is it usefull to keep the 'line view' button on the right since there is no other view possibility ?

[RomuDeuxfois]

Better to use ListItemButton

[Archidoit]

If we click again on 'Extends this search to indexed files', it would be great that the panel closes.

[Archidoit]

Sorting file result (by name or 'attached entity type' is not working :

Note: Would be great to also be able to sort by the number of occurences
@Jipegien

[Archidoit]

What are all the files extensions for which this feature is working ?
I uploaded a txt file containing the world 'filter' and it was not found... But an other more simple txt file was found...

[SouadHadjiat]

What are all the files extensions for which this feature is working ?

@Archidoit We support these mime types : text/plain, text/csv, application/pdf, application/vnd.ms-excel, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet (excel sheets). you can find them in default.json

[SouadHadjiat]

Sorting file result (by name or 'attached entity type' is not working :

Note: Would be great to also be able to sort by the number of occurences

It's a mistake, we don't support custom sorting for now (it will be in V2), we forgot to remove column sorting. And sorting by number of occurrence won't be possible, because it's a data we compute, not something stored that we could sort on.