[Backport 7.53.x] Added checkpoint quantum firewall integration (#17306)

* Added checkpoint quantum firewall integration (#16522)

* Added checkpoint quantum firewall integration

* Add images and update manifest

* Update CHANGELOG.md

* Minor dashboard update

* Update dashboards and folder stucture as per syslog integration

* Moved logs in data collected section

* review comment changes

Co-authored-by: Austin Lai <[email protected]>

* code review dashboard changes

Co-authored-by: Austin Lai <[email protected]>

* Readme changes

* Address review comments

* Update checkpoint_quantum_firewall/README.md

Co-authored-by: Austin Lai <[email protected]>

* Add logs YAML to generate test results

* Update logs test samples

* Rename logs test file

* Update the changelog

* Update log yaml id

* Fix log pipeline and facets failures

* Add `backend_only: false` to logs YAML

* Rename `checkpoint_quantum_firewall.yml` to `checkpoint_quantum_firewall.yaml`

* Add results to log samples

* Update IP addresses in sample for geoip parser

* Add results of geoip parser

* Update checkpoint_quantum_firewall/changelog.d/16522.changed

* Update manifest and add standard attributes in pipeline yaml file

* Fix indentation for samples in yaml test file

* Fix indentation for yaml test file

* Reformat log samples

* Remove test results from test yaml

* Update log samples with service

* Add |- to stringify samples

* Revert |- change as it's returning invalid results

* Update source to replace _ with - and related changes

* Update spec.yaml file

* Add integration name in labeler

* Update in labeler yml file

* Update service in test yaml file

* Update test yaml file

* Add standard attributes in pipeline yaml

* Address review comments

* Update test results

* Update pipeline to rename the attribute path and update dashboards

* Update test results

* Address PR comments

* Updated dashbboards, facets and pipelines

Signed-off-by: nisargshah_crest <[email protected]>

* Updating test results

Signed-off-by: nisargshah_crest <[email protected]>

* correcting results typo

Signed-off-by: nisargshah_crest <[email protected]>

* Updating dashboard and pipeline

Signed-off-by: nisargshah_crest <[email protected]>

* Adding test results

Signed-off-by: nisargshah_crest <[email protected]>

* Updating facet names and dashboards

Signed-off-by: nisargshah_crest <[email protected]>

* Updating test results

Signed-off-by: nisargshah_crest <[email protected]>

* Adding status facet

Signed-off-by: nisargshah_crest <[email protected]>

* Update checkpoint_quantum_firewall/datadog_checks/checkpoint_quantum_firewall/__about__.py

* Removed checkpoint.quantum.firewall.status facet

Signed-off-by: nisargshah_crest <[email protected]>

* updated test results

Signed-off-by: nisargshah_crest <[email protected]>

* Address PR review comments

* Address PR review comments

* Revert changes for changelog

---------

Signed-off-by: nisargshah_crest <[email protected]>
Co-authored-by: Bhargav Nariyani <[email protected]>
Co-authored-by: Austin Lai <[email protected]>
Co-authored-by: mohittilala_crest <[email protected]>
Co-authored-by: Florent Clarret <[email protected]>
Co-authored-by: Florent Clarret <[email protected]>
Co-authored-by: nisargshah_crest <[email protected]>
Co-authored-by: Inigo Mediavilla Saiz <[email protected]>
(cherry picked from commit f9b17b5)

* Nudge CI

---------

Co-authored-by: Tirthraj Chaudhari <[email protected]>

.github/workflows/config/labeler.yml

@@ -89,6 +89,8 @@ integration/ceph:
 - ceph/**/*
 integration/cert_manager:
 - cert_manager/**/*
+integration/checkpoint_quantum_firewall:
+- checkpoint_quantum_firewall/**/*
 integration/cilium:
 - cilium/**/*
 integration/cisco_aci:

checkpoint_quantum_firewall/CHANGELOG.md

@@ -0,0 +1,3 @@
+# CHANGELOG - checkpoint_quantum_firewall
+
+<!-- towncrier release notes start -->

checkpoint_quantum_firewall/README.md

@@ -0,0 +1,140 @@
+## Overview
+
+[Check Point Next Generation Firewall][7] is a security gateway that includes application control and IPS protection, with integrated management of security events. Additional features include Identity Awareness, URL Filtering, Anti-Bot, Anti-Virus, and Anti-Spam.
+
+This integration ingests URL Filtering logs, Anti Bot logs, Application Control, Firewall, Identity Awareness, IPS, Threat Emulation, and miscellaneous event types with the integration log pipeline to enrich the logs and normalizes data to Datadog standard attributes. This integration offers dashboard visualizations with detailed insights into allowed or blocked URLs, bot details, insights into accessed application data, events generated by firewall, mapping between computer identities and machine IP address, and more.
+
+## Setup
+
+### Installation
+
+To install the Checkpoint Quantum Firewall integration, follow the steps below:
+
+**Note**: This step is not necessary for Agent version >= 7.52.0.
+
+1. [Install][5] the 1.0 release (`checkpoint_quantum_firewall==1.0.0`).
+
+### Configuration
+
+#### Log collection
+
+**Checkpoint Quantum Firewall:**
+
+1. Collecting logs is disabled by default in the Datadog Agent. Enable it in the `datadog.yaml` file:
+
+   ```yaml
+   logs_enabled: true
+   ```
+
+2. Add this configuration block to your `checkpoint_quantum_firewall.d/conf.yaml` file to start collecting your Checkpoint Quantum Firewall logs.
+
+   See the [sample checkpoint_quantum_firewall.d/conf.yaml][6] for available configuration options.
+
+   ```yaml
+   logs:
+     - type: tcp/udp
+       port: <PORT>
+       service: checkpoint-quantum-firewall
+       source: checkpoint-quantum-firewall
+   ```
+
+3. [Restart the Agent][1].
+
+4. Configure Syslog Message Forwarding from Checkpoint Quantum Firewall:
+   1. Connect to the command line on the Management Server / Log Server.
+   2. Login to the Expert mode. Enter your administrative credentials (after entering credentials, expert mode is enabled).
+   3. In order to configure a new target for the exported logs, enter the following commands:
+      ```yaml
+      cp_log_export add name <Name of Log Exporter Configuration> target-server <HostName or IP address of Target Server> target-port <Port on Target Server> protocol {tcp | udp} format json
+      ```
+      - In the commands above, specify the following Syslog Server Details:
+
+        - name: The Name of the syslog server. For example: `datadog_syslog`.
+        - target-server: The destination where you want to send the Checkpoint Quantum Firewall logs.
+        - target-port: The port on which the syslog server is listening (typically 514).
+        - protocol: The protocol name, or which protocol will be used to send logs (TCP/UDP).
+        - format: Format must be 'json'.
+   4. In order to save and add the syslog server configuration, use the following command:
+      ```yaml
+      cp_log_export restart name <Name of Log Exporter Configuration>
+      ```
+   5. For more information about configuring syslog, see the [official Checkpoint documentation][4].
+
+### Validation
+
+[Run the Agent's status subcommand][2] and look for `checkpoint_quantum_firewall` under the Checks section.
+
+## Data Collected
+
+### Logs
+
+The Checkpoint Quantum Firewall integration collects Firewall, URL Filtering, IPS, Identity Awareness, Application Control, Threat Emulation, Audit, Anti Ransomware, Anti Spam & Email Security, Anti Exploit, Anti Bot, Anti Virus, HTTPS Inspection, DLP, and Anti Malware logs.
+
+### Metrics
+
+The Checkpoint Quantum Firewall integration does not include any metrics.
+
+### Events
+
+The Checkpoint Quantum Firewall integration does not include any events.
+
+### Service Checks
+
+The Checkpoint Quantum Firewall integration does not include any service checks.
+
+## Troubleshooting
+
+**Checkpoint Quantum Firewall:**
+
+#### Permission denied while port binding
+
+If you see a **Permission denied** error while port binding in the Agent logs, see the following instructions:
+
+1.  Binding to a port number under 1024 requires elevated permissions. Follow the instructions below to set this up.
+
+    - Grant access to the port using the `setcap` command:
+
+      ```
+      sudo setcap CAP_NET_BIND_SERVICE=+ep /opt/datadog-agent/bin/agent/agent
+      ```
+
+    - Verify the setup is correct by running the `getcap` command:
+
+      ```
+      sudo getcap /opt/datadog-agent/bin/agent/agent
+      ```
+
+      With the expected output:
+
+      ```
+      /opt/datadog-agent/bin/agent/agent = cap_net_bind_service+ep
+      ```
+
+      **Note**: Re-run this `setcap` command every time you upgrade the Agent.
+
+2.  [Restart the Agent][1].
+
+#### Data is not being collected
+
+Make sure that traffic is bypassed from the configured port if the firewall is enabled.
+
+#### Port already in use
+
+If you see the **Port <PORT-NO\> Already in Use** error, see the following instructions. The example below is for PORT-NO = 514:
+
+On systems using Syslog, if the Agent listens for Checkpoint Quantum Firewall logs on port 514, the following error can appear in the Agent logs: `Can't start UDP forwarder on port 514: listen udp :514: bind: address already in use`.
+
+This error occurs because by default, Syslog listens on port 514. To resolve this error, take **one** of the following steps:
+
+- Disable Syslog
+- Configure the Agent to listen on a different, available port
+
+For further assistance, contact [Datadog support][3].
+
+[1]: https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent
+[2]: https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information
+[3]: https://docs.datadoghq.com/help/
+[4]: https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_LoggingAndMonitoring_AdminGuide/Content/Topics-LMG/Log-Exporter-Configuration-in-CLI-Basic.htm?tocpath=Log%20Exporter%7CConfiguring%20Log%20Exporter%20in%20CLI%7C_____1
+[5]: https://docs.datadoghq.com/agent/guide/integration-management/?tab=linux#install
+[6]: https://github.com/DataDog/integrations-core/blob/master/checkpoint_quantum_firewall/datadog_checks/checkpoint_quantum_firewall/data/conf.yaml.example
+[7]: https://www.checkpoint.com/quantum/next-generation-firewall/