Investigations Missing Related Infrastructure

[explorecti]

Description

When a user attempts to add additonal objects to a large investigation the new objects are not added.

Example: When creating an investigation and adding 3080 infrastructure SDO's there is no issue. However when selecting all of the infrastructure SDO's and attempting to expand by adding another 2590 related infrastructure SDO's only a small amount of the 2590 infrastructure objects were added.

Environment

1. OS: RHEL8
2. OpenCTI version: 6.1.10
3. OpenCTI client: Frontend

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Navigate to investigations.
2. Create a test investigation and add 3080 or greater infrastructure SDO's.
3. Now select all of the infrastructure SDO's and expand by adding another 2590 or greater related infrastructure
4. Only a small amount of 2590 infrastructure objects were added

Expected Output

All the new objects should be added.

Actual Output

Only 2590 infrastructure objects were added.

Additional information

Maybe there is a threshold on related infrastructure objects.

[nino-filigran]

I'm currently trying to reproduce, but I do not have enough infra so I'm doing it with other entities. Quick question though: why do you need to load this much data in an investigation? What's action you're taking afterwards? Indeed, navigating and manipulating this amount of data does not seem really pratical!

[explorecti]

@nino-filigran This is common for analysis of large amounts of data that is ingested into the application. Investigations are widely utilized and a great tool for an analyst. If the larger amounts of investigation is taxing the application that would be reasonable to deny expanding data object tolerances. If this is not the case it would be helpful if the investigations would allow for more data objects.

[nino-filigran]

Thanks for your answer, I agree on the fact that if we have a limit, we should prevent to display.

I'm still interested in understanding what's the final use case you're trying to achieve, by having this number of entities displayed. Indeed, pivoting from entities to entites is easy when you don't have that much, but with this amount, it does not seem practical. Even to just select a subset is hard.

Are you trying ot find some correlations between them that would be invisible at first sight? If so, based on which criteria?
cc @romain-filigran

[explorecti]

The best I can tell you is that there are many data objects that are being compared and examined that can grow these investigations for our team. In addition the investigations help find multi-hop infrastructure connections between 2 or more Threat Actors/Malware. If there is a limit that is currently set that could be shared this might be helpful. If there is an expansion possible in the future that could be helpful as well.

[nino-filigran]

The whole reason behind my comment is we're working on a specific solution to be able to correlate large amount of data between each others, based on several criteria, such as amount of IOCs/Observables in common, type of common relationships... And I wanted to discover whether or not it's something you're trying to do with your investigation currently, which is not practical right now.

I have not yet got a proper answer regarding amount of elements that can be displayed. When I get it, I'll also share it here.

[explorecti]

Understood, any information you can provide would be helpful.

[nino-filigran]

@explorecti sorry for the time it toook to treat this issue.
OVerall, this is a deep issue that we have in the plateform that we need to solutioned. So I've created a feature to tackle it: #8125

Also, be aware that we have this EPIC #7448 that lists all issues related to graph. The overall approach for to impove graph will be detailed there (it includes the above ticket).