Merge branch 'develop' into shadowserver-dynamic-config

CHANGELOG.md

@@ -19,6 +19,10 @@
     if `auth_by_ssl_client_certificate` is *false*);
   - `password` (STOMP authentication passcode, default: "guest"; to be used only
     if `auth_by_ssl_client_certificate` is *false*).
+- Add the possibility to set the `ssl_ca_certificate` configuration parameter for
+  `intelmq.bots.collectors.stomp.collector` and/or `intelmq.bots.outputs.stomp.output`
+  to an empty string - which means that the SSL machinery used for STOMP communication
+  will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).
 
 ### Core
 - `intelmq.lib.message`: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
@@ -27,7 +31,7 @@
 - `intelmq.lib.mixins`: Add a new class, `StompMixin` (defined in a new submodule: `stomp`),
   which provides certain common STOMP-bot-specific operations, factored out from
   `intelmq.bots.collectors.stomp.collector` and `intelmq.bots.outputs.stomp.output`
-  (PR#2408 by Jan Kaliszewski).
+  (PR#2408 and PR#2414 by Jan Kaliszewski).
 
 ### Development
 - Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
@@ -36,11 +40,16 @@
 
 ### Bots
 #### Collectors
-- `intelmq.bots.collectors.stomp.collector` (PR#2408 by Jan Kaliszewski):
-  - Add support for authentication based on STOMP login and passcode,
-    introducing 3 new configuration parameters (see above: *Configuration*).
+- `intelmq.bots.collectors.stomp.collector` (PR#2408 and PR#2414 by Jan Kaliszewski):
+  - Drop support for versions of `stomp.py` older than `4.1.12`.
   - Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`);
     fixes [#2342](https://github.com/certtools/intelmq/issues/2342).
+  - Add support for authentication based on STOMP login and passcode, introducing three
+    new configuration parameters (see above: *Configuration*).
+  - Add support for loading the system’s default CA certificates, as an alternative to
+    specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
+  - Fix (by carefully targeted monkey patching) certain security problems caused by
+    SSL-related weaknesses that some versions of `stomp.py` suffer from.
   - Fix the reconnection behavior: do not attempt to reconnect after `shutdown`. Also,
     never attempt to reconnect if the version of `stomp.py` is older than `4.1.21` (it
     did not work properly anyway).
@@ -54,29 +63,40 @@
 #### Parsers
 
 #### Experts
+- `intelmq.bots.experts.jinja` (PR#2417 by Mikk Margus Möll):
+  - Add optional `socket_perms` and `socket_group` parameters to change
+    file permissions on socket file, if it is in use.
 
 #### Outputs
-- `intelmq.bots.outputs.stomp.output` (PR#2408 by Jan Kaliszewski):
-  - Add support for authentication based on STOMP login and passcode,
-    introducing 3 new configuration parameters (see above: *Configuration*).
+- `intelmq.bots.outputs.stomp.output` (PR#2408 and PR#2414 by Jan Kaliszewski):
+  - Drop support for versions of `stomp.py` older than `4.1.12`.
   - Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`).
+  - Add support for authentication based on STOMP login and passcode, introducing three
+    new configuration parameters (see above: *Configuration*).
+  - Add support for loading the system’s default CA certificates, as an alternative to
+    specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
+  - Fix (by carefully targeted monkey patching) certain security problems caused by
+    SSL-related weaknesses that some versions of `stomp.py` suffer from.
   - Fix `AttributeError` caused by attempts to get unset attributes of `StompOutputBot`
     (`ssl_ca_cert` et consortes).
   - Add coercion of the `port` config parameter to `int`.
   - Add implementation of the `check` hook (verifying, in particular, accessibility
     of necessary file(s)).
-  - Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.8`).
+  - Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.12`).
   - Minor fixes/improvements and some refactoring (see also above: *Core*...).
 
 ### Documentation
 - Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
 - Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
 - Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s
-  Stream API (PR#2408 by Jan Kaliszewski).
+  Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
 - Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
 
 ### Packaging
 - Add `pendulum` to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
+- `debian/control`: in `Suggests` field, replace ``python3-stomp.py (>= 4.1.9)`` with
+  ``python3-stomp (>= 4.1.12)``, i.e., fix the package name by removing the `.py`
+  suffix and bump the minimum version to `4.1.12` (PR#2414 by Jan Kaliszewski).
 
 ### Tests
 

debian/control

@@ -53,7 +53,7 @@ Suggests: python3-geoip2 (>= 2.2.0),
           python3-pyasn (>= 1.5.0),
           python3-pymongo (>= 2.7.1),
           python3-sleekxmpp (>= 1.3.1),
-          python3-stomp.py (>= 4.1.9),
+          python3-stomp (>= 4.1.12),
           python3-pendulum
 Description: Solution for IT security teams for collecting and processing security feeds
  IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse

docs/index.md

@@ -7,7 +7,7 @@
 [![CII Badge](https://bestpractices.coreinfrastructure.org/projects/4186/badge)](https://bestpractices.coreinfrastructure.org/projects/4186/)
 -->
 
-![IntelMQ](static/images/Logo_Intel_MQ.svg)
+![IntelMQ](/docs/static/images/Logo_Intel_MQ.svg)
 
 # Introduction
 
@@ -60,4 +60,4 @@ It follows the following basic meta-guidelines:
 [^1]: [Incident Handling Automation Project](https://www.enisa.europa.eu/activities/cert/support/incident-handling-automation), mailing list: [email protected]
 
 
-![CEF](https://ec.europa.eu/inea/sites/default/files/ceflogos/en_horizontal_cef_logo_2.png)
+![CEF](https://ec.europa.eu/inea/sites/default/files/ceflogos/en_horizontal_cef_logo_2.png)

docs/user/bots.md

@@ -259,6 +259,14 @@ used. Requires the [tornado](https://pypi.org/project/tornado/) library.
 
 (optional, string) Location of the socket. Defaults to `/tmp/imq_api_default_socket`.
 
+**`socket_perms`**
+
+(optional, octal integer) Unix permissions to grant to the socket file. Default: `600`
+
+**`socket_group`**
+
+(optional, string) Name of group to change group ownership of socket file to.
+
 ---
 
 ### Generic URL Fetcher <div id="intelmq.bots.collectors.http.collector_http" />
@@ -1191,35 +1199,49 @@ Install the `stomp.py` library from PyPI:
 pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt
 ```
 
+Alternatively, you may want to install it using your OS's native
+packaging tools, e.g.:
+
+```bash
+apt install python3-stomp
+```
+
+Apart from that, depending on what STOMP server you connect to, you may
+need to obtain, from the organization or company owning the server, one
+or more of the following security/authentication-related resources:
+
+* CA certificate file;
+* either: *client certificate* and *client certificate's key* files,
+  or: *username* (STOMP *login*) and *password* (STOMP *passcode*).
+
+Also, you will need to know an appropriate STOMP *destination* (aka
+*exchange point*), e.g. `/exchange/my.example.org/*.*.*.*`.
+
 **Parameters (also expects [feed parameters](#feed-parameters)):**
 
 **`server`**
 
-(required, string) Hostname of the STOMP server.
+(required, string) STOMP server's hostname or IP, e.g. "n6stream.cert.pl" (which is default)
 
 **`port`**
 
-(optional, integer) Defaults to 61614.
+(optional, integer) STOMP server's port number (default: 61614)
 
 **`exchange`**
 
-(required, string) STOMP *destination* to subscribe to, e.g. "/exchange/my.org/*.*.*.*"
-
-**`username`**
-
-(optional, string) Username to use.
+(required, string) STOMP *destination* to subscribe to, e.g. `"/exchange/my.org/*.*.*.*"`
 
-**`password`**
+**`heartbeat`**
 
-(optional, string) Password to use.
+(optional, integer) default: 6000
 
 **`ssl_ca_certificate`**
 
-(optional, string) Path to trusted CA certificate.
+(optional, string) Path to CA file, or empty string to load system's default CA certificates
 
 **`auth_by_ssl_client_certificate`**
 
-(optional, boolean) Whether to authenticate using TLS certificate. (Set to false for new *n6* auth.) Defaults to true.
+(optional, boolean) Default: true (note: false is needed for new *n6* auth)
 
 **`ssl_client_certificate`**
 
@@ -1229,6 +1251,14 @@ pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt
 
 (optional, string) Path to client private key to use for TLS connections.
 
+**`username`**
+
+(optional, string) Username to use.
+
+**`password`**
+
+(optional, string) Password to use.
+
 ---
 
 ### Twitter (REMOVE?) <div id="intelmq.bots.collectors.twitter.collector_twitter" />
@@ -5075,72 +5105,87 @@ This bot pushes data to any STOMP stream. STOMP stands for Streaming Text Orient
 
 **Requirements**
 
-Install the stomp.py library, e.g. [apt install python3-stomp.py] or [pip install stomp.py].
+Install the `stomp.py` library from PyPI:
 
-You need a CA certificate, client certificate and key file from the organization / server you are connecting to. Also
-you will need a so called "exchange point".
+```bash
+pip3 install -r intelmq/bots/outputs/stomp/REQUIREMENTS.txt
+```
 
-**Parameters:**
+Alternatively, you may want to install it using your OS's native
+packaging tools, e.g.:
 
-**`exchange`**
+```bash
+apt install python3-stomp
+```
 
-(optional, string) The exchange to push to. Defaults to `/exchange/_push`.
+Apart from that, depending on what STOMP server you connect to, you may
+need to obtain, from the organization or company owning the server, one
+or more of the following security/authentication-related resources:
 
-**`username`**
+* CA certificate file;
+* either: *client certificate* and *client certificate's key* files,
+  or: *username* (STOMP *login*) and *password* (STOMP *passcode*).
 
-(optional, string) Username to use.
+Also, you will need to know an appropriate STOMP *destination* (aka
+*exchange point*), e.g. `/exchange/_push`.
 
-**`password`**
+**Parameters:**
 
-(optional, string) Password to use.
+**`server`**
 
-**`ssl_ca_certificate`**
+(optional, string) STOMP server's hostname or IP, e.g. "n6stream.cert.pl" or "127.0.0.1" (which is default)
 
-(optional, string) Path to trusted CA certificate.
+**`port`**
 
-**`auth_by_ssl_client_certificate`**
+(optional, integer) STOMP server's port number (default: 61614)
+
+**`exchange`**
 
-(optional, boolean) Whether to authenticate using TLS certificate. (Set to false for new *n6* auth.) Defaults to true.
+(optional, string) STOMP *destination* to push at, e.g. ``"/exchange/_push"`` (which is default)
 
 **`heartbeat`**
 
 (optional, integer) Defaults to 60000.
 
-**`message_hierarchical_output`**
+**`ssl_ca_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, string) path to CA file, or empty string to load system's default CA certificates
 
-**`message_jsondict_as_string`**
+**`auth_by_ssl_client_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, boolean) default: true (note: false is needed for new *n6* auth)
 
-**`message_with_type`**
+**`ssl_client_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, string) Path to client certificate to use for TLS connections.
 
-**`port`**
+**`ssl_client_certificate_key`**
 
-(optional, integer) Defaults to 61614.
+(optional, string) Path to client private key to use for TLS connections.
 
-**`server`**
+**`username`**
 
-(optional, string) Hostname of the STOMP server.
+(optional, string) STOMP *login* (e.g., *n6* user login), used only if `auth_by_ssl_client_certificate` is false
 
-**`single_key`**
+**`password`**
 
-(optional, string) Output only a single specified key. In case of `raw` key the data is base64 decoded. Defaults to null (output the whole message).
+(optional, string) STOMP *passcode* (e.g., *n6* user API key), used only if `auth_by_ssl_client_certificate` is false
 
-**`ssl_ca_certificate`**
+**`message_hierarchical_output`**
 
-(optional, string) Path to trusted CA certificate.
+(optional, boolean) Defaults to false.
 
-**`ssl_client_certificate`**
+**`message_jsondict_as_string`**
 
-(optional, string) Path to client certificate to use for TLS connections.
+(optional, boolean) Defaults to false.
 
-**`ssl_client_certificate_key`**
+**`message_with_type`**
 
-(optional, string) Path to client private key to use for TLS connections.
+(optional, boolean) Defaults to false.
+
+**`single_key`**
+
+(optional, string) Output only a single specified key. In case of `raw` key the data is base64 decoded. Defaults to null (output the whole message).
 
 ---
 

docs/user/feeds.md

@@ -719,15 +719,15 @@ parameters:
 
 ### N6 Stomp Stream
 
-N6 Collector - CERT.pl's N6 Collector - N6 feed via STOMP interface. Note that rate_limit does not apply for this bot as it is waiting for messages on a stream.
+N6 Collector - CERT.pl's *n6* Stream API feed (via STOMP interface). Note that 'rate_limit' does not apply to this bot, as it is waiting for messages on a stream.
 
 **Public:** no
 
-**Revision:** 2023-09-23
+**Revision:** 2023-10-08
 
 **Documentation:** <https://n6.readthedocs.io/usage/streamapi/>
 
-**Additional Information:** Contact cert.pl to get access to the feed.
+**Additional Information:** Contact CERT.pl to get access to the feed. Note that the configuration parameter values suggested here are suitable for the new *n6* Stream API variant (with authentication based on 'username' and 'password'); for this variant, typically you can leave the 'ssl_ca_certificate' parameter's value empty - then the system's default CA certificates will be used; however, if that does not work, you need to set 'ssl_ca_certificate' to the path to a file containing CA certificates eligible to verify "*.cert.pl" server certificates (to be found among the publicly available CA certs distributed with modern web browsers/OSes). Also, note that the 'server' parameter's value (for the *new API variant*) suggested here, "n6stream-new.cert.pl", is a temporary domain; ultimately, it will be changed back to "stream.cert.pl". When it comes to the *old API variant* (turned off in November 2023!), you need to have the 'server' parameter set to the name "n6stream.cert.pl", 'auth_by_ssl_client_certificate' set to true, 'ssl_ca_certificate' set to the path to a file containing the *n6*'s legacy self-signed CA certificate (which is stored in file "intelmq/bots/collectors/stomp/ca.pem"), and the parameters 'ssl_client_certificate' and 'ssl_client_certificate_key' set to the paths to your-*n6*-client-specific certificate and key files (note that the 'username' and 'password' parameters are then irrelevant and can be omitted).
 
 
 **Collector configuration**
@@ -736,14 +736,14 @@ N6 Collector - CERT.pl's N6 Collector - N6 feed via STOMP interface. Note that r
 module: intelmq.bots.collectors.stomp.collector
 parameters:
   auth_by_ssl_client_certificate: False
-  exchange: {insert your exchange point as given by CERT.pl}
+  exchange: {insert your STOMP *destination* to subscribe to, as given by CERT.pl, e.g. /exchange/my.example.org/*.*.*.*}
   name: N6 Stomp Stream
-  password: {insert n6 user's API key}
+  password: {insert your *n6* API key}
   port: 61614
   provider: CERT.PL
-  server: n6stream.cert.pl
-  ssl_ca_certificate: {insert path to CA file for CERT.pl's n6}
-  username: {insert n6 user's login}
+  server: n6stream-new.cert.pl
+  ssl_ca_certificate:
+  username: {insert your *n6* login, e.g. [email protected]}
 ```
 
 **Parser configuration**

intelmq/bots/collectors/api/collector_api.py

@@ -8,6 +8,7 @@
 """
 from threading import Thread
 from typing import Optional
+import grp
 import os
 import socket
 
@@ -42,6 +43,8 @@ class APICollectorBot(CollectorBot):
     _is_multithreadable: bool = False
     use_socket = False
     socket_path = '/tmp/imq_api_default_socket'
+    socket_perms = '600'
+    socket_group = ''
     _server: Optional['HTTPServer'] = None
     _unix_socket: Optional[socket.socket] = None
     _eventLoopThread: Optional[Thread] = None
@@ -56,7 +59,12 @@ def init(self):
 
         if self.use_socket:
             self.server = HTTPServer(app)
-            self._unix_socket = bind_unix_socket(self.socket_path)
+            self._unix_socket = bind_unix_socket(self.socket_path, mode=int(self.socket_perms, 8))
+            if self.socket_group:
+                group = grp.getgrnam(self.socket_group)
+                gid = group.gr_gid
+                os.chown(self.socket_path, -1, gid)
+
             self.server.add_socket(self._unix_socket)
         else:
             self.server = app.listen(self.port)

intelmq/bots/collectors/stomp/REQUIREMENTS.txt

@@ -1,4 +1,4 @@
 # SPDX-FileCopyrightText: 2017 Sebastian Wagner
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
-stomp.py>=4.1.8
+stomp.py>=4.1.12