[azure,o365,microsoft_defender_endpoint,m365_defender] Standardize ECS in Microsoft Integrations #13931

mohitjha-elastic · 2025-05-19T12:07:23Z

Proposed Commit Message

azure,m365_defender,microsoft_defender_endpoint,o365: set some ECS fields to standarize the names across
all the microsoft integrations.

* azure: set `service.id`, `device.id`, `user.id`, `session.id`, and `token.id` in graphactivitylogs dataset.
* m365_defender: set `device.id` ECS in all the dataset and `application.name` in event dataset.
* microsoft_defender_endpoint: set device.id from microsoft_defender_endpoint.machine.aad_device_id.
* o365: set `application.name`, `device.id`, `session.id`, and `token.id` in audit dataset.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

To test azure integration

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/azure directory.
Run the following command to run tests.
elastic-package test -v

To test m365_defender integration

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/m365_defender directory.
Run the following command to run tests.
elastic-package test -v

To test microsoft_defender_endpoint integration

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/microsoft_defender_endpoint directory.
Run the following command to run tests.
elastic-package test -v

To test o365 integration

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/o365 directory.
Run the following command to run tests.
elastic-package test -v

Related issues

Part of [Azure]: Standardize Azure field names across all integrations #13369

…standarize_names_in_microsoft_integrations Conflicts: packages/o365/changelog.yml

elasticmachine · 2025-05-19T12:07:28Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-05-19T12:32:12Z

🚀 Benchmarks report

Package `microsoft_defender_endpoint` 👍(0) 💚(0) 💔(3)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`log`	3508.77	1398.6	-2110.17 (-60.14%)	💔
`machine`	2336.45	1438.85	-897.6 (-38.42%)	💔
`machine_action`	7042.25	5882.35	-1159.9 (-16.47%)	💔

To see the full report comment with /test benchmark fullreport

efd6

Suggest commit message:

azure,microsoft_defender_endpoint,o365: map source-specific fields to ECS fields

* azure: *.properties.service_principal_id and *.properties.c_device_id to
  service.id and device.id respectively
* microsoft_defender_endpoint: microsoft_defender_endpoint.machine.aad_device_id
  to device.id
* o365: o365.audit.AppAccessContext.DeviceId to device.id

packages/azure/changelog.yml

packages/microsoft_defender_endpoint/changelog.yml

packages/o365/changelog.yml

kcreddy · 2025-05-20T09:08:50Z

packages/azure/changelog.yml

@@ -1,3 +1,8 @@
+- version: "1.24.0"
+  changes:
+    - description: Map `service.id` and `device.id` ECS fields to the `*.properties.service_principal_id` and `*.properties.c_device_id` fields, respectively.


Looks like there are more fields defined in the requirement: #13369 (comment) for o365 and azure integrations such as user.*, application.*, session.*.

We have only picked up the data streams and integrations that are owned by the security team. One such example is graphactivity data stream (from Azure). During this process, we attempted to map all relevant fields to the ECS schema. However, fields such as application.* and session.* are not part of the current ECS schema and therefore could not be mapped.

I think the ask was to manually define these fields (if not present in ECS). Example. Also if you don't have sample data to populate these fields inside tests, please request them in the issue so they can DM you.

...es/microsoft_defender_endpoint/data_stream/machine/elasticsearch/ingest_pipeline/default.yml

Update changelog descriptions as per suggestions.

…standarize_names_in_microsoft_integrations Conflicts: packages/azure/changelog.yml packages/azure/manifest.yml

efd6 · 2025-05-20T22:44:57Z

There is a confused GH user out there somewhere; my handle is not capitalised.

efd6

LGTM, but there are outstanding concerns from @kcreddy, so waiting for him.

Set service.id, device.id, user.id, session.id and token.id in graphactivitylogs dataset in azure. Set device.id ECS in all the dataset and application.name in event dataset in m365_defender. Set application.name, device.id, session.id, and token.id in audit dataset in o365.

…standarize_names_in_microsoft_integrations Conflicts: packages/m365_defender/changelog.yml packages/m365_defender/data_stream/alert/sample_event.json packages/m365_defender/data_stream/incident/sample_event.json packages/m365_defender/docs/README.md packages/microsoft_defender_endpoint/changelog.yml packages/microsoft_defender_endpoint/data_stream/machine/sample_event.json packages/microsoft_defender_endpoint/docs/README.md

…standarize_names_in_microsoft_integrations Conflicts: packages/o365/changelog.yml packages/o365/manifest.yml

kcreddy · 2025-05-26T11:34:38Z

packages/azure/data_stream/graphactivitylogs/fields/standard-fields.yml

Since we already have base-fields.yml this name might be confusing.
Can you rename the file to ecs-extended.yml and we can understand slightly better as there are RFCs in ECS adding them in the future.

Similarly for other data streams.

packages/azure/docs/graphactivitylogs.md

packages/m365_defender/changelog.yml

packages/o365/docs/README.md

zmoog

I see the PR is security-focused.

Approving on behalf of the elastic/obs-ds-hosted-services team based on co-ownership of packages/azure/(changelog|manifest).yml and packages/azure/docs/.

…standarize_names_in_microsoft_integrations Conflicts: packages/m365_defender/changelog.yml packages/m365_defender/data_stream/alert/sample_event.json packages/m365_defender/data_stream/incident/sample_event.json packages/m365_defender/docs/README.md packages/microsoft_defender_endpoint/changelog.yml packages/microsoft_defender_endpoint/data_stream/log/sample_event.json packages/microsoft_defender_endpoint/docs/README.md

Add description for the fields. Update standard-fields file name to extended-ecs

elasticmachine · 2025-05-27T11:06:01Z

💚 Build Succeeded

Buildkite Build
Commit: 049d94b

History

💚 Build #26391 succeeded 5875dc5
💔 Build #26390 failed 855b69c
💚 Build #26136 succeeded 9040e73
💔 Build #26135 failed d3900fb
💚 Build #26066 succeeded dd770f4

cc @mohitjha-elastic

elastic-sonarqube · 2025-05-27T11:06:10Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
84.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-05-27T13:01:55Z

Package azure - 1.24.0 containing this change is available at https://epr.elastic.co/package/azure/1.24.0/

elastic-vault-github-plugin-prod · 2025-05-27T13:01:56Z

Package m365_defender - 3.7.0 containing this change is available at https://epr.elastic.co/package/m365_defender/3.7.0/

elastic-vault-github-plugin-prod · 2025-05-27T13:01:56Z

Package microsoft_defender_endpoint - 2.37.0 containing this change is available at https://epr.elastic.co/package/microsoft_defender_endpoint/2.37.0/

elastic-vault-github-plugin-prod · 2025-05-27T13:01:57Z

Package o365 - 2.17.0 containing this change is available at https://epr.elastic.co/package/o365/2.17.0/

mohitjha-elastic added 2 commits May 19, 2025 17:17

Standarize ECS fields in Microsoft Integrations

4838693

Merge branch 'main' of github.com:mohitjha-elastic/integrations into …

98526ef

…standarize_names_in_microsoft_integrations Conflicts: packages/o365/changelog.yml

mohitjha-elastic self-assigned this May 19, 2025

mohitjha-elastic requested review from a team as code owners May 19, 2025 12:07

Update changelog entry.

dd770f4

mohitjha-elastic changed the title ~~azure/o365/microsoft_defender_endpoint: Standarize ECS in Microsoft Integrations~~ azure/o365/microsoft_defender_endpoint: Standardize ECS in Microsoft Integrations May 19, 2025

mohitjha-elastic mentioned this pull request May 19, 2025

[Azure]: Standardize Azure field names across all integrations #13369

Open

efd6 reviewed May 19, 2025

View reviewed changes

packages/azure/changelog.yml Outdated Show resolved Hide resolved

packages/microsoft_defender_endpoint/changelog.yml Outdated Show resolved Hide resolved

packages/o365/changelog.yml Outdated Show resolved Hide resolved

kcreddy reviewed May 20, 2025

View reviewed changes

mohitjha-elastic added 2 commits May 20, 2025 17:37

Resolve comments by @Edf6.

d3900fb

Update changelog descriptions as per suggestions.

Merge branch 'main' of github.com:mohitjha-elastic/integrations into …

9040e73

…standarize_names_in_microsoft_integrations Conflicts: packages/azure/changelog.yml packages/azure/manifest.yml

mohitjha-elastic requested review from kcreddy and efd6 May 20, 2025 12:12

efd6 reviewed May 20, 2025

View reviewed changes

mohitjha-elastic added 3 commits May 26, 2025 16:00

Merge branch 'main' of github.com:mohitjha-elastic/integrations into …

5875dc5

…standarize_names_in_microsoft_integrations Conflicts: packages/o365/changelog.yml packages/o365/manifest.yml

mohitjha-elastic changed the title ~~azure/o365/microsoft_defender_endpoint: Standardize ECS in Microsoft Integrations~~ azure/o365/microsoft_defender_endpoint/m365_defender: Standardize ECS in Microsoft Integrations May 26, 2025

kcreddy reviewed May 26, 2025

View reviewed changes

andrewkroh added the Integration:m365_defender Microsoft M365 Defender label May 26, 2025

zmoog approved these changes May 27, 2025

View reviewed changes

mohitjha-elastic added 2 commits May 27, 2025 15:29

Resolve review comment.

049d94b

Add description for the fields. Update standard-fields file name to extended-ecs

mohitjha-elastic changed the title ~~azure/o365/microsoft_defender_endpoint/m365_defender: Standardize ECS in Microsoft Integrations~~ [azure,o365,microsoft_defender_endpoint,m365_defender] Standardize ECS in Microsoft Integrations May 27, 2025

mohitjha-elastic requested a review from kcreddy May 27, 2025 10:43

kcreddy approved these changes May 27, 2025

View reviewed changes

kcreddy merged commit d058c47 into elastic:main May 27, 2025
8 checks passed

[azure,o365,microsoft_defender_endpoint,m365_defender] Standardize ECS in Microsoft Integrations #13931

[azure,o365,microsoft_defender_endpoint,m365_defender] Standardize ECS in Microsoft Integrations #13931

Conversation

mohitjha-elastic commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Commit Message

Checklist

How to test this PR locally

To test azure integration

To test m365_defender integration

To test microsoft_defender_endpoint integration

To test o365 integration

Related issues

Uh oh!

elasticmachine commented May 19, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package microsoft_defender_endpoint 👍(0) 💚(0) 💔(3)

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy May 20, 2025

Choose a reason for hiding this comment

Uh oh!

mohitjha-elastic May 20, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy May 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efd6 commented May 20, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

kcreddy May 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zmoog left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 27, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented May 27, 2025

Quality Gate passed

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented May 27, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented May 27, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented May 27, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented May 27, 2025

Uh oh!

Uh oh!

mohitjha-elastic commented May 19, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented May 19, 2025 •

edited

Loading

Package `microsoft_defender_endpoint` 👍(0) 💚(0) 💔(3)

zmoog left a comment •

edited

Loading