Map 'azure.activitylogs.Level' field to 'log.level'

[muthu-mps]

• Bug

Proposed commit message

This issue fixes the mapping for azure.activitylogs.Level. This value should be populated into log.level instead it is currently pushed to azure.activitylogs.level.

As Azure emits same logs in different field names for same log type we need to process the logs to appropriate field.

Marking the azure.activitylogs.level field as deprecated.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Install the Azure activity logs integration and validate all the log levels are captured in log.level.

Related issues

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[zmoog]

We fixed the azure.activitylogs.level problem by deprecating and removing it from the document.

However, the current values for the log.level are mixed:

cat packages/azure/data_stream/activitylogs/_dev/test/pipeline/test-*expected* | jq '.expected[].log.level'
"Information"
"4"
"4"
"4"
"Information"

I believe we should add a mapping step to map the numeric value "4" to the equivalent textual value, so we can always have values like Critical, Error, Warning, Informational and Verbose.

[muthu-mps]

We fixed the azure.activitylogs.level problem by deprecating and removing it from the document.

However, the current values for the log.level are mixed:

cat packages/azure/data_stream/activitylogs/_dev/test/pipeline/test-*expected* | jq '.expected[].log.level'
"Information"
"4"
"4"
"4"
"Information"

I believe we should add a mapping step to map the numeric value "4" to the equivalent textual value, so we can always have values like Critical, Error, Warning, Informational and Verbose.

Thanks @zmoog! I have mapped the string equivalent values for the severity levels in integer.

[elastic-sonarqube]

Quality Gate failed

Failed conditions
43.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 05e5dca

History

• 💚 Build #26242 succeeded 6444688
• 💚 Build #26237 succeeded bac5a3e

cc @muthu-mps

[zmoog]

There is still a small inconsistency between the textual values and the mapped ones:

cat packages/azure/data_stream/activitylogs/_dev/test/pipeline/test-*expected* | jq '.expected[].log.level'
"Information"
"Informational"
"Informational"
"Informational"
"Information"

Where did you find the mapping between number > text values? I tried for some time but couldn't identify a source I would fully trust.

[muthu-mps]

Thanks for highlighting on the severity levels @zmoog!

• The severity level is listed here in the official document. We need to process for both Information and Informational.
• The text to number values is taken from syslog, log4j etc.. which users standard level to represent their priority or importance, where lower numbers signify higher severity. There is no document which specifies the severity levels mapped to the numbers in Azure.

[zmoog]

I was looking for a doc that explicitly states that 0: Level X, 1: Level Y, and so on. But I could not find one as well.

I see you added "Verbose", even if it's not included at https://learn.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-schema#severity-level

How confident are you we can apply syslog, log4j, etc logic to activity logs?

---

Since all *-raw.log file in the _dev/test/pipeline use "Information", why did you decided to pick "Informational" instead of "Information"? As a user, I'm confused by seeing both "Informational" and "Information" values emitted from the same piece of software, unless there's a clear difference between the two.

Looking through the codebase, I see the other log categories (functions and spring cloud logs) use "Informational" as value in a level field.

I am collecting activity logs for the upcoming EDOT Cloud Forwarder project, so I searched through the samples I have, and this is the result:

➜ cat samples/logs/ds/activity/Policy/eventhub.ndjson | jq '.records[].level' | sort | uniq -c
 210 "Information"
  84 "Warning"

➜ cat samples/logs/ds/activity/Recommendation/eventhub.ndjson | jq '.records[].level' | sort | uniq -c
   2 "Informational"

➜ cat samples/logs/ds/activity/ResourceHealth/eventhub.ndjson | jq '.records[].level' | sort | uniq -c
   5 "Informational"

➜ cat samples/logs/ds/activity/ServiceHealth/eventhub.ndjson | jq '.records[].level' | sort | uniq -c
   1 "Informational"

➜ cat samples/logs/ds/activity/Administrative/eventhub.ndjson | jq '.records[].level' | sort | uniq -c
   4 "Information"

We get a different value depending on the log category 🤦

[zmoog]

For the "Informational" vs. "Information", the only things I can think of are:

1. Update the docs saying that each service emits its own version of this severity level, suggesting users to add their own custom pipeline to adjust values.
2. Add an optional setting to allow users to override this level with the value they prefer.

[zmoog]

"Informational" vs. "Information": it seems there's no way out, we need to decide which poison to pick.

Mapping number/text: I would love to find a official doc, or piece of code in a SDK, with a reliable mapping to double-check our assumptions.

[efd6]

Suggested change

	- description: Map `azure.activitylogs.Level` field to log.level.
	- description: Map `azure.activitylogs.Level` field to `log.level`.