Crowdstrike FDR: Scaling host metadata enrichment

[chemamartinez]

As part of #2816, a cache processor was added to enrich FDR events with host and user metadata at ingest-time.

That means that right now, the cached metadata is stored locally in the agent and the enrichment doesn't work when agents are scaled horizontally.

This issue is intended to track our progress on researching possible solutions/workarounds.

Ideas

• Support storing data in memcached or redis (something that is available as a service on CSPs). Make the existing cache processor "multi-layer" with read-through to the distributed cache when the local memory cache doesn't contain the key.

• ES|QL is adding a new lookup join feature that could be used to perform the metadata join at query time. That would simplify the architecture as it doesn't require any changes on the agent side. See [Discuss] Supporting ES|QL LOOKUP JOIN on integration data package-spec#873.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)