Skip to content

azure: fix Grok processor error for firewall network rule logs #13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

JulienOrain
Copy link
Contributor

Proposed commit message

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@JulienOrain JulienOrain requested review from a team as code owners May 15, 2025 15:06
@andrewkroh andrewkroh added Integration:azure Azure Logs Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels May 15, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@efd6
Copy link
Contributor

efd6 commented May 15, 2025

/test

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after nit is addressed.

@@ -8,3 +8,4 @@
{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"HTTP request from 192.168.0.2:54314 to ocsp.sca1b.amazontrust.com:80. Url: ocsp.sca1b.amazontrust.com. Action: Deny. ThreatIntel: Bot Networks"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}
{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"ICMP request from 192.168.0.2: to 175.16.199.1:. Action: alert. Signature: 2100366. IDS: ICMP_INFO PING *NIX. Priority: 3. Classification: Misc activity"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}
{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389. Action: Allow.. Rule Collection: Permit_RFC1918. Rule: Permit_RFC1918"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2025-03-13T07:11:59.992099+00:00"}
{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNatRuleLog","properties":{"msg":"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Rule Collection: DNAT. Rule: rule"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the missing final new line.

@efd6
Copy link
Contributor

efd6 commented May 15, 2025

Please run elastic-package test pipeline -d firewall-logs -g.

test case failed: Expected results are different from actual ones: --- want
+++ got
@@ -896,7 +896,7 @@
                     "network"
                 ],
                 "kind": "event",
-                "original": "{\"category\":\"AzureFirewallNetworkRule\",\"operationName\":\"AzureFirewallNatRuleLog\",\"properties\":{\"msg\":\"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Policy: policy-01. Rule Collection Group: DefaultDnatRuleCollectionGroup. Rule Collection: DNAT. Rule: rule\"},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2022-06-08T20:40:56.4525380Z\"}",
+                "original": "{\"category\":\"AzureFirewallNetworkRule\",\"operationName\":\"AzureFirewallNatRuleLog\",\"properties\":{\"msg\":\"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Rule Collection: DNAT. Rule: rule\"},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2022-06-08T20:40:56.4525380Z\"}",
                 "type": [
                     "connection"
                 ]

@JulienOrain
Copy link
Contributor Author

Run pipeline tests for the package
--- Test results for package: azure - START ---
╭─────────┬───────────────┬───────────┬─────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM   │ TEST TYPE │ TEST NAME                                                           │ RESULT │ TIME ELAPSED │
├─────────┼───────────────┼───────────┼─────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-applicationrules-raw.log)            │ PASS   │ 338.244791ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-applicationrules-structured-raw.log) │ PASS   │ 301.347125ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-dnsproxy-structured-raw.log)         │ PASS   │ 294.812875ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-dnsproxyrules-raw.log)               │ PASS   │ 301.480917ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-natrule-structured-raw.log)          │ PASS   │ 302.623875ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-networkrule-structured-raw.log)      │ PASS   │    288.899ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-networkrules-raw.log)                │ PASS   │ 321.231334ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-sdh3075-raw.log)                     │ PASS   │  313.15875ms │
│ azure   │ firewall_logs │ pipeline  │ test-applicationrules-raw.log                                       │ PASS   │ 135.278834ms │
│ azure   │ firewall_logs │ pipeline  │ test-applicationrules-structured-raw.log                            │ PASS   │  69.822334ms │
│ azure   │ firewall_logs │ pipeline  │ test-dnsproxy-structured-raw.log                                    │ PASS   │   69.52575ms │
│ azure   │ firewall_logs │ pipeline  │ test-dnsproxyrules-raw.log                                          │ PASS   │     87.385ms │
│ azure   │ firewall_logs │ pipeline  │ test-natrule-structured-raw.log                                     │ PASS   │  68.191667ms │
│ azure   │ firewall_logs │ pipeline  │ test-networkrule-structured-raw.log                                 │ PASS   │  68.047167ms │
│ azure   │ firewall_logs │ pipeline  │ test-networkrules-raw.log                                           │ PASS   │ 161.161041ms │
│ azure   │ firewall_logs │ pipeline  │ test-sdh3075-raw.log                                                │ PASS   │  64.328209ms │
╰─────────┴───────────────┴───────────┴─────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: azure - END   ---
Done

@efd6
Copy link
Contributor

efd6 commented May 16, 2025

/test

@elastic-vault-github-plugin-prod

🚀 Benchmarks report

Package azure 👍(5) 💚(4) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
auditlogs 2386.63 2016.13 -370.5 (-15.52%) 💔
platformlogs 4901.96 2673.8 -2228.16 (-45.45%) 💔

To see the full report comment with /test benchmark fullreport

Copy link

@elasticmachine
Copy link

💚 Build Succeeded

History

@@ -1,3 +1,8 @@
- version: "1.23.3"
changes:
- description: Fix Grok processor error in ingest pipeline for `AzureFirewallNetworkRuleLog` in `azure.firewall_logs`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- description: Fix Grok processor error in ingest pipeline for `AzureFirewallNetworkRuleLog` in `azure.firewall_logs`.
- description: Add missing pattern for DNAT request messages for `AzureFirewallNetworkRuleLog` in firewall_logs data stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Integration:azure Azure Logs Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants