[Entity Analytics] [Entity Store] [API] Changes to support event.ingested as a configurable timestamp field for init and enable endpoints

[abhishekbhatia1710]

Summary

This PR introduces support for configuring the timestamp field for entity store enablement.

By default, the timestamp field is set to @timestamp, but users can opt to use event.ingested or another preferred value based on their requirements.

Entity Store API changes

Entity Store enable

POST kbn:/api/entity_store/enable
{
  "timestampField": "event.ingested"
}

Result

{
  "engines": [
    {
      "status": "started",
      "type": "user",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "event.ingested"
    },
    {
      "status": "started",
      "type": "host",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "event.ingested"
    },
    {
      "status": "started",
      "type": "universal",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "event.ingested"
    }
  ],
  "status": "running"
}

POST kbn:/api/entity_store/enable
{
}

Result

{
  "engines": [
    {
      "status": "started",
      "type": "host",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    },
    {
      "status": "started",
      "type": "user",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    },
    {
      "status": "started",
      "type": "universal",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    }
  ],
  "status": "running"
}

Different entity types can have distinct timestampField values, as described below:

For instance, the host entity uses event.ingested as its timestampField, while other entities default to the @timestamp field.

POST kbn:/api/entity_store/engines/host/init
{
  "timestampField": "event.ingested"
}

POST kbn:/api/entity_store/engines/user/init
{
}

Result

{
  "engines": [
    {
      "status": "started",
      "type": "universal",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    },
    {
      "status": "started",
      "type": "host",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "event.ingested"
    },
    {
      "status": "started",
      "type": "user",
      "indexPattern": "",
      "filter": "",
      "fieldHistoryLength": 10,
      "timestampField": "@timestamp"
    }
  ],
  "status": "running"
}

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Testing steps

• Check out this PR branch.
• Start Kibana.
• Avoid enabling the Entity Store from the UI initially. Instead, enable it via the API by following the steps outlined below.

DELETE kbn:/api/entity_store/engines/user
DELETE kbn:/api/entity_store/engines/host

GET kbn:/api/entity_store/status

 All engines with the same timestamp Field
POST kbn:/api/entity_store/enable
{
  "timestampField": "event.ingested"
}

# All engines with the same timestamp Field
POST kbn:/api/entity_store/enable
{
}

# Different entity with different timestamp Field
POST kbn:/api/entity_store/engines/host/init
{
  "timestampField": "event.ingested"
}

POST kbn:/api/entity_store/engines/user/init
{
}

4. Afterward, you can test the functionality through the UI. First, clear the entity data from the UI, then start the Entity Store. Note that the UI currently does not offer a way to configure this parameter.

[elasticmachine]

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

[yctercero]

Hey there! Just passed by this in my email and just a thought - curious if we'll want to think about a more global setting for overriding timestamp. Not sure if the use case is very different, but rules also allow an override. I imagine at some point, setting overrides for every individual feature might be cumbersome. Nothing to block this effort there, just dropping a thought :)

cc @jaredburgettelastic

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[abhishekbhatia1710]

Hi @yctercero! That’s a great point, and this is being discussed internally. A more global setting for timestamp overrides could definitely help streamline configurations across multiple features, preventing duplication and ensuring consistency.

Right now, the focus is on enabling configurability at the entity store level, but this could be a good direction for a broader discussion, perhaps aligning it with how rules handle overrides to maintain uniformity across the system.

Appreciate you bringing this up! Definitely something worth considering as this functionality evolves. 🚀

[jaredburgettelastic]

@elasticmachine merge upstream

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 8142565

Failed CI Steps

• Jest Tests #20

Test Failures

• [job] [logs] Jest Tests #20 / Connector renders loading state correctly

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	21.4MB	21.4MB	+443.0B

History

• 💔 Build #271445 failed a26f522
• 💔 Build #271427 failed e0d4dc0
• 💔 Build #271416 failed 0df7975
• 💔 Build #271375 failed e7ef2de
• 💔 Build #270054 failed 4b118da

cc @abhishekbhatia1710

[hop-dev]

@jaredburgettelastic was it a requirement to add a config value for this? I thought we were avoiding this going forward.

[jaredburgettelastic]

Agreed, let's strip this out.

[TinaHeiligers]

LGTM

[jaredburgettelastic]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation