Skip to content

azure: fix Grok processor error for firewall network rule logs #13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

azure: fix Grok processor error for firewall network rule logs #13920

wants to merge 5 commits into from

Conversation

JulienOrain
Copy link
Contributor

Proposed commit message

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@JulienOrain JulienOrain requested review from a team as code owners May 15, 2025 15:06
@andrewkroh andrewkroh added Integration:azure Azure Logs Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels May 15, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@efd6
Copy link
Contributor

efd6 commented May 15, 2025

/test

efd6
efd6 reviewed May 15, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after nit is addressed.

packages/azure/data_stream/firewall_logs/_dev/test/pipeline/test-networkrules-raw.log Outdated
@@ -8,3 +8,4 @@
{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"HTTP request from 192.168.0.2:54314 to ocsp.sca1b.amazontrust.com:80. Url: ocsp.sca1b.amazontrust.com. Action: Deny. ThreatIntel: Bot Networks"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}
{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"ICMP request from 192.168.0.2: to 175.16.199.1:. Action: alert. Signature: 2100366. IDS: ICMP_INFO PING *NIX. Priority: 3. Classification: Misc activity"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}
{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389. Action: Allow.. Rule Collection: Permit_RFC1918. Rule: Permit_RFC1918"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2025-03-13T07:11:59.992099+00:00"}
{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNatRuleLog","properties":{"msg":"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Rule Collection: DNAT. Rule: rule"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the missing final new line.

@efd6
Copy link
Contributor

efd6 commented May 15, 2025

Please run elastic-package test pipeline -d firewall-logs -g.

test case failed: Expected results are different from actual ones: --- want
+++ got
@@ -896,7 +896,7 @@
                     "network"
                 ],
                 "kind": "event",
-                "original": "{\"category\":\"AzureFirewallNetworkRule\",\"operationName\":\"AzureFirewallNatRuleLog\",\"properties\":{\"msg\":\"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Policy: policy-01. Rule Collection Group: DefaultDnatRuleCollectionGroup. Rule Collection: DNAT. Rule: rule\"},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2022-06-08T20:40:56.4525380Z\"}",
+                "original": "{\"category\":\"AzureFirewallNetworkRule\",\"operationName\":\"AzureFirewallNatRuleLog\",\"properties\":{\"msg\":\"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Rule Collection: DNAT. Rule: rule\"},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2022-06-08T20:40:56.4525380Z\"}",
                 "type": [
                     "connection"
                 ]

@JulienOrain
Copy link
Contributor Author 

Run pipeline tests for the package
--- Test results for package: azure - START ---
╭─────────┬───────────────┬───────────┬─────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM   │ TEST TYPE │ TEST NAME                                                           │ RESULT │ TIME ELAPSED │
├─────────┼───────────────┼───────────┼─────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-applicationrules-raw.log)            │ PASS   │ 338.244791ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-applicationrules-structured-raw.log) │ PASS   │ 301.347125ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-dnsproxy-structured-raw.log)         │ PASS   │ 294.812875ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-dnsproxyrules-raw.log)               │ PASS   │ 301.480917ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-natrule-structured-raw.log)          │ PASS   │ 302.623875ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-networkrule-structured-raw.log)      │ PASS   │    288.899ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-networkrules-raw.log)                │ PASS   │ 321.231334ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-sdh3075-raw.log)                     │ PASS   │  313.15875ms │
│ azure   │ firewall_logs │ pipeline  │ test-applicationrules-raw.log                                       │ PASS   │ 135.278834ms │
│ azure   │ firewall_logs │ pipeline  │ test-applicationrules-structured-raw.log                            │ PASS   │  69.822334ms │
│ azure   │ firewall_logs │ pipeline  │ test-dnsproxy-structured-raw.log                                    │ PASS   │   69.52575ms │
│ azure   │ firewall_logs │ pipeline  │ test-dnsproxyrules-raw.log                                          │ PASS   │     87.385ms │
│ azure   │ firewall_logs │ pipeline  │ test-natrule-structured-raw.log                                     │ PASS   │  68.191667ms │
│ azure   │ firewall_logs │ pipeline  │ test-networkrule-structured-raw.log                                 │ PASS   │  68.047167ms │
│ azure   │ firewall_logs │ pipeline  │ test-networkrules-raw.log                                           │ PASS   │ 161.161041ms │
│ azure   │ firewall_logs │ pipeline  │ test-sdh3075-raw.log                                                │ PASS   │  64.328209ms │
╰─────────┴───────────────┴───────────┴─────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: azure - END   ---
Done

@efd6
Copy link
Contributor

efd6 commented May 16, 2025

/test

@elastic-vault-github-plugin-prod
Copy link

🚀 Benchmarks report

Package azure 👍(5) 💚(4) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
auditlogs 2386.63 2016.13 -370.5 (-15.52%) 💔
platformlogs 4901.96 2673.8 -2228.16 (-45.45%) 💔

To see the full report comment with /test benchmark fullreport

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
50.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link

💚 Build Succeeded

History

efd6
efd6 reviewed May 18, 2025
View reviewed changes
packages/azure/changelog.yml
@@ -1,3 +1,8 @@
- version: "1.23.3"
changes:
- description: Fix Grok processor error in ingest pipeline for `AzureFirewallNetworkRuleLog` in `azure.firewall_logs`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- description: Fix Grok processor error in ingest pipeline for `AzureFirewallNetworkRuleLog` in `azure.firewall_logs`.
- description: Add missing pattern for DNAT request messages for `AzureFirewallNetworkRuleLog` in firewall_logs data stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@efd6 efd6 efd6 left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
Integration:azure Azure Logs Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@JulienOrain @elasticmachine @efd6 @andrewkroh