diff --git a/crowdsec-docs/docs/configuration/crowdsec_configuration.md b/crowdsec-docs/docs/configuration/crowdsec_configuration.md index ea38b918..ecf46e8e 100644 --- a/crowdsec-docs/docs/configuration/crowdsec_configuration.md +++ b/crowdsec-docs/docs/configuration/crowdsec_configuration.md @@ -150,6 +150,7 @@ db_config: flush: max_items: "" max_age: "" + metrics_max_age: "" bouncers_autodelete: cert: "" api_key: "" @@ -192,6 +193,11 @@ api: #- 127.0.0.1 #- ::1 #- 10.0.0.0/24 + auto_registration: + enabled: + token: + allowed_ranges: + - 10.0.0.0/24 prometheus: enabled: "(true|false)" level: "(full|aggregated)" @@ -447,6 +453,7 @@ db_config: flush: max_items: "" max_age: "" + metrics_max_age: "" bouncers_autodelete: cert: "" api_key: "" @@ -593,6 +600,7 @@ When WAL is unspecified you will see the following warning message at startup : flush: max_items: max_age: + metrics_max_age: bouncers_autodelete: cert: "" api_key: "" @@ -611,6 +619,21 @@ Number max of alerts in database. Alerts retention time. +Supported units: + + - `s`: seconds + + - `m`: minutes + + - `h`: hours + + - `d`: days + +#### `metrics_max_age` +> string + +Usage metrics retention time. + Supported units: - `s`: seconds @@ -688,7 +711,11 @@ api: - bouncers_ou crl_path: "" cache_expiration: "" - + auto_registration: + enabled: + token: + allowed_ranges: + - 10.0.0.0/24 ``` #### `cti` @@ -790,6 +817,11 @@ server: - bouncers_ou crl_path: "" cache_expiration: "" + auto_registration: + enabled: + token: + allowed_ranges: + - 10.0.0.0/24 ``` ##### `enable` @@ -935,6 +967,40 @@ The format must be compatible with golang [time.Duration](https://pkg.go.dev/tim IPs or IP ranges which have admin access to API. The APIs would still need to have API keys. 127.0.0.1 and ::1 are always given admin access whether specified or not. +#### `auto_registration` + +This section configures LAPI to automatically accept new machine registrations + +```yaml +auto_registration: + enabled: + token: + allowed_ranges: + - 10.0.0.0/24 +``` + +##### `enabled` +> bool + +Whether automatic registration should be enabled. + +Defaults to `false`. + +##### `token` +> string + +Token that should be passed in the registration request if LAPI needs to automatically validate the machine. + +It must be at least 32 chars, and is mandatory if the feature is enabled. + +##### `allowed_ranges` +> []string + +IP ranges that are allowed to use the auto registration features. + +It must have at least one entry if the feature is enabled + + ### `prometheus` This section is used by local API and crowdsec. diff --git a/crowdsec-docs/docs/cscli/cscli_bouncers.md b/crowdsec-docs/docs/cscli/cscli_bouncers.md index a2dc2e13..469eb79f 100644 --- a/crowdsec-docs/docs/cscli/cscli_bouncers.md +++ b/crowdsec-docs/docs/cscli/cscli_bouncers.md @@ -36,6 +36,7 @@ Note: This command requires database direct access, so is intended to be run on * [cscli](/cscli/cscli.md) - cscli allows you to manage crowdsec * [cscli bouncers add](/cscli/cscli_bouncers_add.md) - add a single bouncer to the database * [cscli bouncers delete](/cscli/cscli_bouncers_delete.md) - delete bouncer(s) from the database +* [cscli bouncers inspect](/cscli/cscli_bouncers_inspect.md) - inspect a bouncer by name * [cscli bouncers list](/cscli/cscli_bouncers_list.md) - list all bouncers within the database * [cscli bouncers prune](/cscli/cscli_bouncers_prune.md) - prune multiple bouncers from the database diff --git a/crowdsec-docs/docs/cscli/cscli_bouncers_delete.md b/crowdsec-docs/docs/cscli/cscli_bouncers_delete.md index 9c719154..76c64418 100644 --- a/crowdsec-docs/docs/cscli/cscli_bouncers_delete.md +++ b/crowdsec-docs/docs/cscli/cscli_bouncers_delete.md @@ -10,10 +10,17 @@ delete bouncer(s) from the database cscli bouncers delete MyBouncerName [flags] ``` +### Examples + +``` +cscli bouncers delete "bouncer1" "bouncer2" +``` + ### Options ``` - -h, --help help for delete + -h, --help help for delete + --ignore-missing don't print errors if one or more bouncers don't exist ``` ### Options inherited from parent commands diff --git a/crowdsec-docs/docs/cscli/cscli_bouncers_inspect.md b/crowdsec-docs/docs/cscli/cscli_bouncers_inspect.md new file mode 100644 index 00000000..d7e6aad9 --- /dev/null +++ b/crowdsec-docs/docs/cscli/cscli_bouncers_inspect.md @@ -0,0 +1,41 @@ +--- +id: cscli_bouncers_inspect +title: cscli bouncers inspect +--- +## cscli bouncers inspect + +inspect a bouncer by name + +``` +cscli bouncers inspect [bouncer_name] [flags] +``` + +### Examples + +``` +cscli bouncers inspect "bouncer1" +``` + +### Options + +``` + -h, --help help for inspect +``` + +### Options inherited from parent commands + +``` + --color string Output color: yes, no, auto (default "auto") + -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") + --debug Set logging to debug + --error Set logging to error + --info Set logging to info + -o, --output string Output format: human, json, raw + --trace Set logging to trace + --warning Set logging to warning +``` + +### SEE ALSO + +* [cscli bouncers](/cscli/cscli_bouncers.md) - Manage bouncers [requires local API] + diff --git a/crowdsec-docs/docs/cscli/cscli_hub_update.md b/crowdsec-docs/docs/cscli/cscli_hub_update.md index c6e041f0..e988cd1c 100644 --- a/crowdsec-docs/docs/cscli/cscli_hub_update.md +++ b/crowdsec-docs/docs/cscli/cscli_hub_update.md @@ -19,7 +19,8 @@ cscli hub update [flags] ### Options ``` - -h, --help help for update + -h, --help help for update + --with-content Download index with embedded item content ``` ### Options inherited from parent commands diff --git a/crowdsec-docs/docs/cscli/cscli_lapi_context.md b/crowdsec-docs/docs/cscli/cscli_lapi_context.md index a7f4014f..ea92cf33 100644 --- a/crowdsec-docs/docs/cscli/cscli_lapi_context.md +++ b/crowdsec-docs/docs/cscli/cscli_lapi_context.md @@ -6,10 +6,6 @@ title: cscli lapi context Manage context to send with alerts -``` -cscli lapi context [command] [flags] -``` - ### Options ``` diff --git a/crowdsec-docs/docs/cscli/cscli_lapi_register.md b/crowdsec-docs/docs/cscli/cscli_lapi_register.md index 378973f1..59abdd3e 100644 --- a/crowdsec-docs/docs/cscli/cscli_lapi_register.md +++ b/crowdsec-docs/docs/cscli/cscli_lapi_register.md @@ -21,6 +21,7 @@ cscli lapi register [flags] -f, --file string output file destination -h, --help help for register --machine string Name of the machine to register with + --token string Auto registration token to use -u, --url string URL of the API (ie. http://127.0.0.1) ``` diff --git a/crowdsec-docs/docs/cscli/cscli_machines.md b/crowdsec-docs/docs/cscli/cscli_machines.md index 6cfca621..9f2e71c4 100644 --- a/crowdsec-docs/docs/cscli/cscli_machines.md +++ b/crowdsec-docs/docs/cscli/cscli_machines.md @@ -42,6 +42,7 @@ cscli machines [action] * [cscli](/cscli/cscli.md) - cscli allows you to manage crowdsec * [cscli machines add](/cscli/cscli_machines_add.md) - add a single machine to the database * [cscli machines delete](/cscli/cscli_machines_delete.md) - delete machine(s) by name +* [cscli machines inspect](/cscli/cscli_machines_inspect.md) - inspect a machine by name * [cscli machines list](/cscli/cscli_machines_list.md) - list all machines in the database * [cscli machines prune](/cscli/cscli_machines_prune.md) - prune multiple machines from the database * [cscli machines validate](/cscli/cscli_machines_validate.md) - validate a machine to access the local API diff --git a/crowdsec-docs/docs/cscli/cscli_machines_delete.md b/crowdsec-docs/docs/cscli/cscli_machines_delete.md index b8b6eda2..4f7060da 100644 --- a/crowdsec-docs/docs/cscli/cscli_machines_delete.md +++ b/crowdsec-docs/docs/cscli/cscli_machines_delete.md @@ -19,7 +19,8 @@ cscli machines delete "machine1" "machine2" ### Options ``` - -h, --help help for delete + -h, --help help for delete + --ignore-missing don't print errors if one or more machines don't exist ``` ### Options inherited from parent commands diff --git a/crowdsec-docs/docs/cscli/cscli_machines_inspect.md b/crowdsec-docs/docs/cscli/cscli_machines_inspect.md new file mode 100644 index 00000000..d3e556f3 --- /dev/null +++ b/crowdsec-docs/docs/cscli/cscli_machines_inspect.md @@ -0,0 +1,42 @@ +--- +id: cscli_machines_inspect +title: cscli machines inspect +--- +## cscli machines inspect + +inspect a machine by name + +``` +cscli machines inspect [machine_name] [flags] +``` + +### Examples + +``` +cscli machines inspect "machine1" +``` + +### Options + +``` + -h, --help help for inspect + -H, --hub show hub state +``` + +### Options inherited from parent commands + +``` + --color string Output color: yes, no, auto (default "auto") + -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") + --debug Set logging to debug + --error Set logging to error + --info Set logging to info + -o, --output string Output format: human, json, raw + --trace Set logging to trace + --warning Set logging to warning +``` + +### SEE ALSO + +* [cscli machines](/cscli/cscli_machines.md) - Manage local API machines [requires local API] + diff --git a/crowdsec-docs/docs/cscli/cscli_notifications.md b/crowdsec-docs/docs/cscli/cscli_notifications.md index da8fe06a..7c90f0a5 100644 --- a/crowdsec-docs/docs/cscli/cscli_notifications.md +++ b/crowdsec-docs/docs/cscli/cscli_notifications.md @@ -32,8 +32,8 @@ To list/inspect/test notification template ### SEE ALSO * [cscli](/cscli/cscli.md) - cscli allows you to manage crowdsec -* [cscli notifications inspect](/cscli/cscli_notifications_inspect.md) - Inspect active notifications plugin configuration -* [cscli notifications list](/cscli/cscli_notifications_list.md) - list active notifications plugins +* [cscli notifications inspect](/cscli/cscli_notifications_inspect.md) - Inspect notifications plugin +* [cscli notifications list](/cscli/cscli_notifications_list.md) - list notifications plugins * [cscli notifications reinject](/cscli/cscli_notifications_reinject.md) - reinject an alert into profiles to trigger notifications * [cscli notifications test](/cscli/cscli_notifications_test.md) - send a generic test alert to notification plugin diff --git a/crowdsec-docs/docs/cscli/cscli_notifications_inspect.md b/crowdsec-docs/docs/cscli/cscli_notifications_inspect.md index 9efb5a89..32461c64 100644 --- a/crowdsec-docs/docs/cscli/cscli_notifications_inspect.md +++ b/crowdsec-docs/docs/cscli/cscli_notifications_inspect.md @@ -4,11 +4,11 @@ title: cscli notifications inspect --- ## cscli notifications inspect -Inspect active notifications plugin configuration +Inspect notifications plugin ### Synopsis -Inspect active notifications plugin and show configuration +Inspect notifications plugin and show configuration ``` cscli notifications inspect [flags] diff --git a/crowdsec-docs/docs/cscli/cscli_notifications_list.md b/crowdsec-docs/docs/cscli/cscli_notifications_list.md index 763a256f..198fa3d2 100644 --- a/crowdsec-docs/docs/cscli/cscli_notifications_list.md +++ b/crowdsec-docs/docs/cscli/cscli_notifications_list.md @@ -4,11 +4,11 @@ title: cscli notifications list --- ## cscli notifications list -list active notifications plugins +list notifications plugins ### Synopsis -list active notifications plugins +list notifications plugins and their status (active or not) ``` cscli notifications list [flags] diff --git a/crowdsec-docs/docs/cscli/cscli_notifications_test.md b/crowdsec-docs/docs/cscli/cscli_notifications_test.md index b08e27b2..b9c30e0c 100644 --- a/crowdsec-docs/docs/cscli/cscli_notifications_test.md +++ b/crowdsec-docs/docs/cscli/cscli_notifications_test.md @@ -8,7 +8,7 @@ send a generic test alert to notification plugin ### Synopsis -send a generic test alert to a notification plugin to test configuration even if is not active +send a generic test alert to a notification plugin even if it is not active in profiles ``` cscli notifications test [plugin name] [flags] diff --git a/crowdsec-docs/docs/cscli/cscli_support_dump.md b/crowdsec-docs/docs/cscli/cscli_support_dump.md index fd44f731..6148f9f5 100644 --- a/crowdsec-docs/docs/cscli/cscli_support_dump.md +++ b/crowdsec-docs/docs/cscli/cscli_support_dump.md @@ -8,20 +8,17 @@ Dump all your configuration to a zip file for easier support ### Synopsis -Dump the following informations: +Dump the following information: - Crowdsec version - OS version -- Installed collections list -- Installed parsers list -- Installed scenarios list -- Installed postoverflows list -- Installed context list -- Bouncers list -- Machines list -- CAPI status -- LAPI status +- Enabled feature flags +- Latest Crowdsec logs (log processor, LAPI, remediation components) +- Installed collections, parsers, scenarios... +- Bouncers and machines list +- CAPI/LAPI status - Crowdsec config (sensitive information like username and password are redacted) - Crowdsec metrics +- Stack trace in case of process crash ``` cscli support dump [flags] diff --git a/crowdsec-docs/docs/notification_plugins/elasticsearch.md b/crowdsec-docs/docs/notification_plugins/elasticsearch.md index 44f46c25..a0b691ed 100644 --- a/crowdsec-docs/docs/notification_plugins/elasticsearch.md +++ b/crowdsec-docs/docs/notification_plugins/elasticsearch.md @@ -3,10 +3,20 @@ id: elastic title: Elasticsearch --- -Elasticsearch can be integrated with CrowdSec by using the HTTP plugin. Enable it by following these [instructions](/notification_plugins/http.md) . +CrowdSec can forward Alerts to Elasticsearch using the HTTP plugin. This guide will show you how to configure the plugin to send alerts to your Elasticsearch instance. + +## Configuring the plugin + +By default the configuration for HTTP plugin is located at these default location per OS: + +- **Linux** `/etc/crowdsec/notifications/http.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` Then replace the `url` and the `format` of the plugin's config so that it posts the events to your Elasticsearch instance. +### Base configuration + An example configuration: ```yaml @@ -29,7 +39,7 @@ headers: ``` -## Authentication +### Authentication If you have enabled security on your elasticsearch cluster, you will have to add a custom `Authorization` header to be able to insert the events. @@ -61,7 +71,7 @@ headers: ``` -## Self-Signed certificate +### Self-Signed certificate If your elasticsearch cluster uses a self-signed certificate, you must set `skip_tls_verification` to `true` in your configuration: ```yaml @@ -84,7 +94,7 @@ headers: ``` -## Potential mapping issues +### Potential mapping issues If you are facing errors because mapper complains about field types inference, ie: @@ -355,6 +365,57 @@ headers: Authorization: "Basic [redacted]" ``` +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. + +``` +#notifications: +# - http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with http plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +
+ ## Final Steps: Let's restart crowdsec diff --git a/crowdsec-docs/docs/notification_plugins/email.md b/crowdsec-docs/docs/notification_plugins/email.md index 33ad4c23..2ff9cc0f 100644 --- a/crowdsec-docs/docs/notification_plugins/email.md +++ b/crowdsec-docs/docs/notification_plugins/email.md @@ -3,27 +3,19 @@ id: email title: Email Plugin --- -The Email plugin is shipped by default with CrowdSec. This guide shows how to enable it. +The Email plugin is shipped by default with CrowdSec. The following guide shows how to configure, test and enable it. -## Enabling the plugin: +## Configuring the plugin -In the profile configuration (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section: +By default the configuration for Email plugin is located at these default location per OS: -``` -#notifications: -# - email_default -``` - -Every alert that passes the profile's filter will be dispatched to the `email_default` plugin. +- **Linux** `/etc/crowdsec/notifications/email.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/email.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\email.yaml` -## Configuring the plugin: +### Base configuration -The default configuration for the email plugin is located at `/etc/crowdsec/notifications/email.yaml`. -You need to provide the credentials for the SMTP server here. - -### Example configuration for Gmail - -Here's an example configuration that sends alerts to `receiver@gmail.com`: +Here is the base configuration for the Email plugin: ```yaml type: email # Don't change @@ -84,6 +76,73 @@ encryption_type: "ssltls" The `format` configuration directive is a [go template](https://pkg.go.dev/text/template), which receives a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. +Typical port and TLS/SSL settings + +| Port | Encryption Type | +|------|-----------------| +| 25 | none | +| 465 | ssltls | +| 587 | starttls | + +:::warning +Port 25 should be avoided at all costs as it is commonly blocked by ISPs and email providers and is insecure as it sends in plain text. +::: + +:::info +Port settings above are common, but may vary depending on your email provider. Please refer to your email provider's documentation for the correct settings. +::: + +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test email_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `email_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `email_default` plugin list item. + +``` +#notifications: +# - email_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `email_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with email plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - email_default +on_success: break +``` + +
+ ## Final Steps: Restart CrowdSec with the following command: @@ -91,5 +150,3 @@ Restart CrowdSec with the following command: ```bash sudo systemctl restart crowdsec ``` - -To verify if the plugin is functioning correctly, you can trigger scenarios using tools like wapiti, nikto etc. diff --git a/crowdsec-docs/docs/notification_plugins/file.md b/crowdsec-docs/docs/notification_plugins/file.md index 4ca26502..d206bf35 100644 --- a/crowdsec-docs/docs/notification_plugins/file.md +++ b/crowdsec-docs/docs/notification_plugins/file.md @@ -3,26 +3,19 @@ id: file title: File Plugin --- -The File plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it. +The File plugin is by default shipped with your CrowdSec installation and allows you to write Alerts to an external file that can be monitored by external applications. The following guide shows how to configure, test and enable it. -## Enabling the plugin: +## Configuring the plugin -In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section +By default the configuration for Email plugin is located at these default location per OS: -``` -#notifications: -# - file_default -``` - -Every alert which would pass the profile's filter would be dispatched to `file_default` plugin. +- **Linux** `/etc/crowdsec/notifications/file.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/file.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\file.yaml` -## Configuring the plugin: +### Base configuration -By default the configuration for File plugin is located at `/etc/crowdsec/notifications/file.yaml`. - -### Adding the plugin configuration - -Example config which writes a ndjson file to `/tmp/crowdsec_alerts.json`. +Example config which writes Alerts to a file using NDJson (**N**ewline **D**elimiter **J**ava**S**cript **O**bject **N**otation) format to `/tmp/crowdsec_alerts.json`. ```yaml # Don't change this @@ -56,9 +49,9 @@ rotate: Some SIEM agents may not support some top level keys we define in the default ndjson format. Please make sure to adjust the format to match your SIEM agent's requirements. ::: -## SIEM Integration +### SIEM Integration -### Filebeat +#### Filebeat Filebeat has a set of reserved top level keys and should not be used in the ndjson format. The following format can be used to be compatible with Filebeat: @@ -69,6 +62,57 @@ format: | {{ end -}} ``` +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test file_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `file_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `file_default` plugin list item. + +``` +#notifications: +# - file_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `file_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with file plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - file_default +on_success: break +``` + +
+ ## Final Steps: Let's restart crowdsec @@ -76,5 +120,3 @@ Let's restart crowdsec ```bash sudo systemctl restart crowdsec ``` - -You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc. \ No newline at end of file diff --git a/crowdsec-docs/docs/notification_plugins/gotify.md b/crowdsec-docs/docs/notification_plugins/gotify.md index 3350f5e0..e35dd33e 100644 --- a/crowdsec-docs/docs/notification_plugins/gotify.md +++ b/crowdsec-docs/docs/notification_plugins/gotify.md @@ -3,11 +3,21 @@ id: gotify title: Gotify --- -Gotify can be integrated with CrowdSec by using the HTTP plugin. Enable it by following these [instructions](/notification_plugins/http.md) . +CrowdSec can forward Alerts to Gotify via the HTTP plugin. This guide will show you how to configure the HTTP plugin to send alerts to your Gotify instance. -Then replace the `` and the `` of the plugin's config so that it send the events to your Gotify instance. +## Configuring the plugin + +By default the configuration for HTTP plugin is located at these default location per OS: + +- **Linux** `/etc/crowdsec/notifications/http.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` + +### Base configuration -An example configuration: +You can replace the file contents with the following configuration: + +Then replace the `` and the `` of the plugin's config so that it send the events to your Gotify instance. ```yaml type: http # Don't change @@ -55,6 +65,57 @@ headers: # skip_tls_verification: # true or false. Default is false ``` +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. + +``` +#notifications: +# - http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with http plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +
+ ## Final Steps: Let's restart crowdsec diff --git a/crowdsec-docs/docs/notification_plugins/helpers.md b/crowdsec-docs/docs/notification_plugins/helpers.md index 7f4f97b8..48dfbd8b 100644 --- a/crowdsec-docs/docs/notification_plugins/helpers.md +++ b/crowdsec-docs/docs/notification_plugins/helpers.md @@ -9,7 +9,7 @@ In order to simplify some operation in the templates, we provide some custom hel The [Sprig](https://masterminds.github.io/sprig/) library is available in the templates, and provides a lot of useful functions. Refer to the sprig documentation for more information. -## Crowdsec specific helpers +## CrowdSec specific helpers ### `HTMLEscape` diff --git a/crowdsec-docs/docs/notification_plugins/http.md b/crowdsec-docs/docs/notification_plugins/http.md index 4cbedb2b..075c6fca 100644 --- a/crowdsec-docs/docs/notification_plugins/http.md +++ b/crowdsec-docs/docs/notification_plugins/http.md @@ -3,26 +3,23 @@ id: http title: HTTP Plugin --- -The HTTP plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it. +The HTTP plugin is by default shipped with your CrowdSec installation. The following guide shows how to configure, test and enable it. -## Enabling the plugin: +Every alert which would pass the profile's filter would be dispatched to `http_default` plugin. -In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section -``` -#notifications: -# - http_default -``` +## Configuring the plugin +By default the configuration for HTTP plugin is located at these default location per OS: -Every alert which would pass the profile's filter would be dispatched to `http_default` plugin. -## Configuring the plugin: +- **Linux** `/etc/crowdsec/notifications/http.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` -By default the configuration for HTTP plugin is located at `/etc/crowdsec/notifications/http.yaml`. Configure how to make web requests by providing the `url`, `method`, `headers` etc. ### Adding the plugin configuration -By default there would be a http config at `/etc/crowdsec/notifications/http.yaml`. Configure how to make web requests by providing the `url`, `method`, `headers` etc. +Configure how to make web requests by providing the `url`, `method`, `headers` etc. Example config which posts the alerts serialized into json to localhost server. @@ -37,6 +34,7 @@ format: | # This template receives list of models.Alert objects. The request bo {{.|toJson}} url: http://localhost # plugin will make requests to this url. Eg value https://www.example.com/ +# unix_socket: /var/run/example.sock # plugin will send the `url` across the unix socket instead of opening a remote connection method: POST # eg either of "POST", "GET", "PUT" and other http verbs is valid value. @@ -55,8 +53,60 @@ method: POST # eg either of "POST", "GET", "PUT" and other http verbs is valid v ``` -**Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. +:::info +`format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. +::: + +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. + +``` +#notifications: +# - http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with http plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +
## Final Steps: @@ -65,5 +115,3 @@ Let's restart crowdsec ```bash sudo systemctl restart crowdsec ``` - -You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc. \ No newline at end of file diff --git a/crowdsec-docs/docs/notification_plugins/intro.md b/crowdsec-docs/docs/notification_plugins/intro.md index df13fe35..df9e5413 100644 --- a/crowdsec-docs/docs/notification_plugins/intro.md +++ b/crowdsec-docs/docs/notification_plugins/intro.md @@ -3,30 +3,25 @@ id: intro title: Introduction --- -import CodeBlock from '@theme/CodeBlock'; - ### Goal -CrowdSec supports notification plugins, meant to be able to push alerts to third party services for alerting or integration purposes. -At the time of writing, plugins exists for [slack](/notification_plugins/slack.md), [splunk](/notification_plugins/splunk.md), and a generic [http push](/notification_plugins/http.md) plugin (allowing to push to services such as [elasticsearch](/notification_plugins/elasticsearch.md)). +CrowdSec supports notification plugins, which allows alerts to pushed to third party services for alerting or integration purposes. -Plugins are defined at LAPI level. Events get dispatched to said plugins via [profile configuration](/profiles/intro.md). +Plugins are defined and used at the LAPI level, so if you are running a multi-server setup, you will configure the plugins on the server that is receiving the alerts. If you are not running a multi-server setup, you will configure the plugins on the same server as the main CrowdSec process. ### Configuration -The default plugins are shipped with CrowdSec upon installation, and can trivially be enabled without further installation. +By default all plugins are shipped with CrowdSec are within the install package, and can trivially be enabled without further need to install additional packages. Refer directly to each plugin's dedicated documentation and keep in mind that plugins needs to be enabled/dispatched at the [profile](/profiles/intro.md) level via the dedicated `notifications` section (defaults to `/etc/crowdsec/profiles.yaml`.md). - - Plugin binaries are present in `config_paths.plugin_dir` (defaults to `/var/lib/crowdsec/plugins/`), and their individual configuration are present in `config_paths.notification_dir` (defaults to `/etc/crowdsec/notifications/`) - -**Important:** CrowdSec rejects the plugins if one of the following is true : +:::warning +CrowdSec rejects the plugins binaries if one of the following is true : 1. plugin is not owned by the root user and root group. 2. plugin is world-writable. - +::: ### Environment variables @@ -68,7 +63,8 @@ notifications: ### Notification plugin configuration: -Following are the fields CrowdSec main process can interpret. +Following fields are provided to all notification plugins and not specific to any plugin. + ```yaml type: name: @@ -77,10 +73,8 @@ group_wait: group_threshold: max_retry: timeout: - - - ``` + #### `type` : Required. Type of plugin, eg "slack" @@ -126,32 +120,36 @@ Currently only `notification` plugins are supported. Whenever CrowdSec receives [See](https://github.com/crowdsecurity/crowdsec/blob/plugins/pkg/protobufs/notifier.proto) the gRPC protocol for `notification` plugins. +In the following sections we use `/etc/crowdsec/config.yaml` for configuration file paths. However depending on your platform the paths can be interchanged with the following: + +- **Linux** `/etc/crowdsec/config.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/config.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\config.yaml` + #### Plugin Discovery Plugins are discovered from the directories specified in `/etc/crowdsec/config.yaml`. -```yaml -#/etc/crowdsec/config.yaml -..... +```yaml title="/etc/crowdsec/config.yaml" config_paths: notification_dir: /etc/crowdsec/notifications/ plugin_dir: /var/lib/crowdsec/plugins/ -..... ``` #### Plugin Process Owner -Due to security reasons, plugins are ideally ran with dropped priveleges. This is done by setting owner and group of the plugin process as some unprivileged user. This can be configured via setting the desired user and group in `/etc/crowdsec/config.yaml`. +Due to security reasons, plugins process are operated under a user with limited privileges. This is done by setting owner and group of the plugin process as some unprivileged user. This can be configured via setting the desired user and group in `/etc/crowdsec/config.yaml`. -```yaml -#/etc/crowdsec/config.yaml -..... +```yaml title="/etc/crowdsec/config.yaml" plugin_config: user: nobody group: nogroup -..... ``` +:::note +Depending on your distribution or platform these values may change to `nobody` or `nogroup`. If you wish to update these values, please ensure that the user and group exist on your system. +::: + ### Alert object You have access to the list of alerts that triggered the notification when writing the go-template in the `format` parameter. @@ -190,8 +188,9 @@ To use them in a go-template, you can check [here](https://pkg.go.dev/github.com
Show the full alert object - -{`[ + +```json +[ { "capacity": 5, "decisions": [ @@ -572,8 +571,9 @@ To use them in a go-template, you can check [here](https://pkg.go.dev/github.com "start_at": "2022-02-12T14:10:21Z", "stop_at": "2022-02-12T14:10:23Z" } -]`} - +] +``` +
#### Usage examples @@ -606,55 +606,19 @@ Extract the meta associated with the alerts {{ end }} ``` ---- - -Teams webhook - -``` - { - "$schema": "https://adaptivecards.io/schemas/adaptive-card.json", - "type": "AdaptiveCard", - "version": "1.0", - "body": [ - {{range .}} - {{ $decisions_len := len .Decisions }} - { - "type": "TextBlock", - "text": "Attack start: {{.StartAt}}" - }, - { - "type": "TextBlock", - "text": "Attack End: {{.StopAt}}" - }, - {{ range $index, $element := .Decisions }} - { - "type": "TextBlock", - "text": "{{$element.Value}} performed {{$element.Scenario}} and got a {{$element.Duration}} {{$element.Type}}" - } - {{ if lt $index (sub $decisions_len 1) }} - , - {{ end }} - {{ end }} - {{end}} - ] - } -``` - ### Debugging notifications plugins -cscli cli tool provide some useful command to help write notification +**cscli** tool provide some useful command to help write notification plugin configuration. Those are provided by the `cscli notifications` command and its subcommands. -First `cscli notifications list` will list the active configured -notifications plugins. Then it's possible to get configuration -information for each notification plugin using its given name by -executing `cscli notifications inspect `. - -The last other helpful subcommand is for testing notifications plugins -directly. It takes an id and reinject the alert through the matched -profile notifications. If the alert matches a profile with no -configured notifications then no notification will be generated. This -can be useful to test both the profile configuration and the normal -operation of the notifications plugins. +| SubCommand | Description | +|---------|-------------| +| [list](/cscli/cscli_notifications_list.md) | List all notification plugins and their status | +| [inspect](/cscli/cscli_notifications_inspect.md) | Get configuration information for a notification plugin | +| [test](/cscli/cscli_notifications_test.md) | Test the configuration by sending a `generic` alert directly to notification plugin | +| [reinject](/cscli/cscli_notifications_reinject.md) | Reinject an Alert to the profiles pipeline to simulate real processing of an Alert | +:::info +Please note the difference between `reinject` and `test`, `reinject` will send the alert to the profiles pipeline and then to the notification plugin that is `active` on the matched profile, while `test` will send the alert directly to the notification plugin no matter if the plugin is active or defined within a profile. +::: diff --git a/crowdsec-docs/docs/notification_plugins/sentinel.md b/crowdsec-docs/docs/notification_plugins/sentinel.md index 91397e39..159a387f 100644 --- a/crowdsec-docs/docs/notification_plugins/sentinel.md +++ b/crowdsec-docs/docs/notification_plugins/sentinel.md @@ -3,21 +3,18 @@ id: sentinel title: Sentinel Plugin --- -The sentinel plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it. +The sentinel plugin is by default shipped with your CrowdSec installation. The following guide shows how to configure, test and enable it. -## Enabling the plugin: +## Configuring the plugin -In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section -``` -#notifications: -# - sentinel_default -``` +By default there would be a sentinel config at these default location per OS: -## Configuring the plugin: +- **Linux** `/etc/crowdsec/notifications/sentinel.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/sentinel.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\sentinel.yaml` -### Adding the plugin configuration +### Base configuration -By default there would be a sentinel config at `/etc/crowdsec/notifications/sentinel.yaml`. You will need to specify: - customer_id - shared_key @@ -70,13 +67,61 @@ The log type is the name of the log that will be sent to azure. Assuming you chose `crowdsec`, it will appear as `crowdsec_CL` in azure. +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test sentinel_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `sentinel_default` with the new name. +::: + +## Enabling the plugin -## Final Steps: +In your profiles you will need to uncomment the `notifications` key and the `sentinel_default` plugin list item. + +``` +#notifications: +# - sentinel_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `sentinel_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with sentinel plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - sentinel_default +on_success: break +``` + +
+ +## Final Steps Let's restart crowdsec ```bash sudo systemctl restart crowdsec ``` - -You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc. \ No newline at end of file diff --git a/crowdsec-docs/docs/notification_plugins/slack.md b/crowdsec-docs/docs/notification_plugins/slack.md index 3ef93883..ec6f66ef 100644 --- a/crowdsec-docs/docs/notification_plugins/slack.md +++ b/crowdsec-docs/docs/notification_plugins/slack.md @@ -5,22 +5,17 @@ title: Slack Plugin The slack plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it. -## Enabling the plugin: - -In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section -``` -#notifications: -# - slack_default -``` - ## Configuring the plugin: -### Adding the plugin configuration +By default the configuration for Slack plugin is located at these default location per OS: + +- **Linux** `/etc/crowdsec/notifications/slack.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/slack.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\slack.yaml` -By default there would be a slack config at `/etc/crowdsec/notifications/slack.yaml`. Specify your -`webhook`. +### Base configuration -Example config: +Here is the base configuration for the Slack plugin: ```yaml # Don't change this @@ -52,6 +47,56 @@ See [slack guide](https://slack.com/intl/en-in/help/articles/115005265063-Incomi **Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test slack_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `slack_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `slack_default` plugin list item. + +``` +#notifications: +# - slack_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `slack_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with email plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - slack_default +on_success: break +``` + +
## Final Steps: @@ -60,5 +105,3 @@ Let's restart crowdsec ```bash sudo systemctl restart crowdsec ``` - -You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc. \ No newline at end of file diff --git a/crowdsec-docs/docs/notification_plugins/splunk.md b/crowdsec-docs/docs/notification_plugins/splunk.md index d845d47e..2360c34c 100644 --- a/crowdsec-docs/docs/notification_plugins/splunk.md +++ b/crowdsec-docs/docs/notification_plugins/splunk.md @@ -5,22 +5,17 @@ title: Splunk Plugin The splunk plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it. -## Enabling the plugin: +## Configuring the plugin: -In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section -``` -#notifications: -# - splunk_default -``` +By default the configuration for Splunk plugin is located at these default location per OS: -## Configuring the plugin: +- **Linux** `/etc/crowdsec/notifications/splunk.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/splunk.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\splunk.yaml` -### Adding the plugin configuration +### Base configuration -By default there would be a splunk config at `/etc/crowdsec/notifications/splunk.yaml`. Specify your -`url`, `token` and `format` . - -Example configuration which posts creates splunk event containing alerts serialized to JSON: +Here is the base configuration for the Splunk plugin: ```yaml # Don't change this @@ -51,6 +46,57 @@ See [splunk guide](https://docs.splunk.com/Documentation/Splunk/8.2.1/Data/Useth **Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test splunk_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `splunk_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `splunk_default` plugin list item. + +``` +#notifications: +# - splunk_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `splunk_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with Splunk plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - splunk_default +on_success: break +``` + +
+ ## Final Steps: Let's restart crowdsec diff --git a/crowdsec-docs/docs/notification_plugins/teams.md b/crowdsec-docs/docs/notification_plugins/teams.md index 286ba7eb..6eaeacf4 100644 --- a/crowdsec-docs/docs/notification_plugins/teams.md +++ b/crowdsec-docs/docs/notification_plugins/teams.md @@ -3,19 +3,17 @@ id: teams title: Microsoft Teams --- -This guide will show you how to enable Microsoft Teams notifications via the HTTP plugin. +The following guide shows how to configure, test and enable HTTP plugin to forward Alerts to Microsoft Teams. -## Enabling the plugin: +## Configuring the plugin -In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section -``` -#notifications: -# - http_default -``` +By default the configuration for HTTP plugin is located at these default location per OS: -## Configuring the plugin: +- **Linux** `/etc/crowdsec/notifications/http.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` -By default there would be a http config at `/etc/crowdsec/notifications/http.yaml`. Simply replace the whole content in this file with this example below. +Simply replace the whole content in this file with this example below. ### Base configuration @@ -291,6 +289,57 @@ headers: * See [microsoft docs](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) for instructions to obtain a webhook. * The `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects. +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. + +``` +#notifications: +# - http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with http plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +
+ ## Final Steps: Let's restart crowdsec @@ -298,5 +347,3 @@ Let's restart crowdsec ```bash sudo systemctl restart crowdsec ``` - -You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc. diff --git a/crowdsec-docs/docs/notification_plugins/telegram.md b/crowdsec-docs/docs/notification_plugins/telegram.md index 9a8800cf..8548e3bb 100644 --- a/crowdsec-docs/docs/notification_plugins/telegram.md +++ b/crowdsec-docs/docs/notification_plugins/telegram.md @@ -3,14 +3,24 @@ id: telegram title: Telegram --- -Telegram can be integrated with CrowdSec by using the HTTP plugin. Enable it by following these [instructions](/notification_plugins/http.md) . +CrowdSec can forward Alerts to telegram via the HTTP plugin. This guide will show you how to configure the HTTP plugin to send alerts to your Telegram chat. + +## Configuring the plugin + +By default the configuration for HTTP plugin is located at these default location per OS: + +- **Linux** `/etc/crowdsec/notifications/http.yaml` +- **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml` +- **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml` + +### Base configuration + +You can replace the file contents with the following configuration: Replace `chat_id` within the format section so that it send the events to your Telegram chat. If you need to get your chat ID, follow the instructions [here](https://stackoverflow.com/questions/32423837/telegram-bot-how-to-get-a-group-chat-id). Replace `XXX:YYY` within the URL section with your Telegram BOT API key. If you need to generate a BOT API key, follow the instructions [here](https://core.telegram.org/bots#how-do-i-create-a-bot). -An example configuration: - ```yaml type: http # Don't change name: http_default # Must match the registered plugin in the profile @@ -67,6 +77,57 @@ headers: Content-Type: "application/json" ``` +## Testing the plugin + +Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin. + +```bash +cscli notifications test http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +## Enabling the plugin + +In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item. + +``` +#notifications: +# - http_default +``` + +:::note +If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name. +::: + +:::warning +Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile. +::: + +
+ +Example profile with http plugin enabled + +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +
+ ## Final Steps: Let's restart crowdsec diff --git a/crowdsec-docs/docs/observability/usage_metrics.md b/crowdsec-docs/docs/observability/usage_metrics.md new file mode 100644 index 00000000..83b57971 --- /dev/null +++ b/crowdsec-docs/docs/observability/usage_metrics.md @@ -0,0 +1,78 @@ +--- +id: usage_metrics +title: Usage Metrics +sidebar_position: 6 +--- + +:::info +Usage metrics require at least CrowdSec v1.6.3 + +Support on Remediation Components are rolling out progressively. Please check the relevant [documentation](/u/bouncers/intro) to see if your Remediation Component has support. +::: + +Logs processors and Remediation Components can provide detailed usage data to the [Local API (LAPI)](local_api/intro.md), allowing for a unified view of their behavior and better insights. + +## Remediation Components + +Remediation Components can send detailed information about themselves and what decisions they are acting on. + +The specific metrics sent vary depending on the type of Remediation Component used. + +For example, the [firewall Remediation Component](/u/bouncers/firewall) can report metrics on dropped bytes or packets, while the [OpenResty Remediation Component](/u/bouncers/openresty) can report metrics on dropped HTTP requests. + +The same applies to interpreting the metrics: when blocking at the firewall level, most bots or attackers stop once they realize they can't connect to the target server. Therefore, the dropped packets or bytes should be seen as relative indicators of effectiveness between different blocking sources, not as the exact number of packets that would have been transmitted if the IP weren't blocked. + +In contrast, HTTP-based Remediation Components typically count each handled request, as attackers are less likely to stop after receiving a 403 response or a Captcha challenge. + +Whenever possible, the Remediation Components will break down the remediated traffic by the source of the decision. + +Currently, CrowdSec supports the following origins: + - `crowdsec`: an automated decision based on behavioral analysis of your logs + - `CAPI`: a decision coming from the Community Blocklist + - `cscli`: a manual decision added with [`cscli decisions add`](cscli/cscli_decisions_add.md) + - `cscli-import`: decisions that were imported with [`cscli decisions import`](cscli/cscli_decisions_import.md) + - `appsec`: the request was blocked by an appsec rule + - `console`: a manual decision added from the [console](https://app.crowdsec.net) + - `lists:XXX`: a decision coming from a blocklist subscribed in the [console](https://app.crowdsec.net). `XXX` is the name of the blocklist. + + +You can view the metrics locally using [`cscli metrics show bouncers`](cscli/cscli_metrics_show.md): + +![usage metrics csli](/img/usage_metrics_cscli_example.png) + +The Remediation Components will send the number of decisions that are actually enforced. + +These numbers may differ from what is shown by [`cscli decisions list`](cscli/cscli_decisions_list.md) for several reasons: +- Filters are applied when querying LAPI (such as scope, scenarios, etc.). +- LAPI deduplicates decisions before sending. If an IP is listed in multiple sources, only the decision with the longest remaining time is sent (useful for assessing blocklist overlap). + +Remediation components will also send the version of the OS they are running on. You can see this information with [`cscli bouncers inspect XXX`](cscli/cscli_bouncers_inspect.md): + +![usage metrics bouncer OS](/img/usage_metrics_bouncer_os.png) + +## Log Processors + +:::info +Log Processors are the underlying component within the Security Engine that processes logs and sends Alerts to the LAPI. If you are running a multi-server setup, you will have multiple Log Processors. +::: + +Logs processors can also send more information about themselves to LAPI: + - Operating system information (version, distribution/platform) + - Number of [datasources](data_sources/introduction.md) configured per type + - Enabled [features flags](configuration/feature_flags.md) + - Installed Hub files (including [custom / tainted](/u/troubleshooting/intro#why-are-some-scenariosparsers-tainted-or-custom-) files): + - AppSec-Config + - AppSec-Rules + - Collections + - Contexts + - Parsers + - Scenarios + + +You can show this data by using [`cscli machines inspect XXX`](cscli/cscli_machines_inspect.md): + +![usage metrics LP](/img/usage_metrics_lp_cscli.png) + +By default, only the collections are shown in order to keep the output readable. + +If you want to see the entire hub state of a given Log Processor, you can use `cscli machines inspect --hub XXX`. \ No newline at end of file diff --git a/crowdsec-docs/docs/parsers/patterns-documentation.md b/crowdsec-docs/docs/parsers/patterns-documentation.md index cb0cac0b..932a8ace 100644 --- a/crowdsec-docs/docs/parsers/patterns-documentation.md +++ b/crowdsec-docs/docs/parsers/patterns-documentation.md @@ -666,7 +666,7 @@ Pattern : Pattern : ``` -[a-zA-Z][a-zA-Z0-9_.+-=:]+ +[a-zA-Z0-9_.+-=:]+ ``` ## URIPATHPARAM diff --git a/crowdsec-docs/docusaurus.config.js b/crowdsec-docs/docusaurus.config.js index 145a9e22..7e08064c 100644 --- a/crowdsec-docs/docusaurus.config.js +++ b/crowdsec-docs/docusaurus.config.js @@ -68,6 +68,7 @@ module.exports = { }, ], themeConfig: { + image: "img/crowdsec_og_image.png", announcementBar: { id: "banner_docs", content: diff --git a/crowdsec-docs/sidebars.js b/crowdsec-docs/sidebars.js index 4262aa63..a0338ecd 100644 --- a/crowdsec-docs/sidebars.js +++ b/crowdsec-docs/sidebars.js @@ -273,6 +273,7 @@ items: [ "observability/cscli", "observability/prometheus", + "observability/usage_metrics", "observability/dashboard", "observability/pprof", ], @@ -412,6 +413,7 @@ items: [ "cscli/cscli_bouncers_add", "cscli/cscli_bouncers_delete", + "cscli/cscli_bouncers_inspect", "cscli/cscli_bouncers_list", "cscli/cscli_bouncers_prune", ], @@ -563,6 +565,7 @@ items: [ "cscli/cscli_machines_add", "cscli/cscli_machines_delete", + "cscli/cscli_machines_inspect", "cscli/cscli_machines_list", "cscli/cscli_machines_prune", "cscli/cscli_machines_validate", diff --git a/crowdsec-docs/sidebarsUnversioned.js b/crowdsec-docs/sidebarsUnversioned.js index e97f2145..c9daa4ba 100644 --- a/crowdsec-docs/sidebarsUnversioned.js +++ b/crowdsec-docs/sidebarsUnversioned.js @@ -329,6 +329,7 @@ module.exports = { "integrations/f5", "integrations/fortinet", "integrations/paloalto", + "integrations/sophos", "integrations/genericfirewall", "integrations/remediationcomponent", ] @@ -350,6 +351,11 @@ module.exports = { id: "troubleshooting/remediation_components", label: "Remediation Components", }, + { + type: "doc", + id: "troubleshooting/cti", + label: "CTI", + }, ], serviceApiSideBar: [ { @@ -454,6 +460,7 @@ module.exports = { "getting_started/post_installation/console_hub", ], }, + "getting_started/post_installation/whitelists", { type: "category", label: "Acquisition", diff --git a/crowdsec-docs/static/img/console/blocklists/org_subscribe_popup.png b/crowdsec-docs/static/img/console/blocklists/org_subscribe_popup.png new file mode 100644 index 00000000..04e38497 Binary files /dev/null and b/crowdsec-docs/static/img/console/blocklists/org_subscribe_popup.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-actions.png b/crowdsec-docs/static/img/console/security_engines/details-page-actions.png index 89487da8..24fece4f 100644 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page-actions.png and b/crowdsec-docs/static/img/console/security_engines/details-page-actions.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-blocklists.png b/crowdsec-docs/static/img/console/security_engines/details-page-blocklists.png index f02cc686..4d78fd9f 100644 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page-blocklists.png and b/crowdsec-docs/static/img/console/security_engines/details-page-blocklists.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-inactive-bouncer.png b/crowdsec-docs/static/img/console/security_engines/details-page-inactive-bouncer.png index c5394f98..25bd38d3 100644 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page-inactive-bouncer.png and b/crowdsec-docs/static/img/console/security_engines/details-page-inactive-bouncer.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-log-processors.png b/crowdsec-docs/static/img/console/security_engines/details-page-log-processors.png index 094afa80..6f9f88c8 100644 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page-log-processors.png and b/crowdsec-docs/static/img/console/security_engines/details-page-log-processors.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-remediation-decisions.png b/crowdsec-docs/static/img/console/security_engines/details-page-remediation-decisions.png new file mode 100644 index 00000000..fa41afaa Binary files /dev/null and b/crowdsec-docs/static/img/console/security_engines/details-page-remediation-decisions.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-remediation-metrics.png b/crowdsec-docs/static/img/console/security_engines/details-page-remediation-metrics.png new file mode 100644 index 00000000..4cf6ced2 Binary files /dev/null and b/crowdsec-docs/static/img/console/security_engines/details-page-remediation-metrics.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-remediation.png b/crowdsec-docs/static/img/console/security_engines/details-page-remediation.png index c86fe65d..2231ee39 100644 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page-remediation.png and b/crowdsec-docs/static/img/console/security_engines/details-page-remediation.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-scenarios-hub.png b/crowdsec-docs/static/img/console/security_engines/details-page-scenarios-hub.png index 14bf9fe8..4a9a3112 100644 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page-scenarios-hub.png and b/crowdsec-docs/static/img/console/security_engines/details-page-scenarios-hub.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-scenarios.png b/crowdsec-docs/static/img/console/security_engines/details-page-scenarios.png index b1f331c3..50100e98 100644 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page-scenarios.png and b/crowdsec-docs/static/img/console/security_engines/details-page-scenarios.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page-summary.png b/crowdsec-docs/static/img/console/security_engines/details-page-summary.png index a516075f..17303a13 100644 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page-summary.png and b/crowdsec-docs/static/img/console/security_engines/details-page-summary.png differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page.jpeg b/crowdsec-docs/static/img/console/security_engines/details-page.jpeg new file mode 100644 index 00000000..b938017a Binary files /dev/null and b/crowdsec-docs/static/img/console/security_engines/details-page.jpeg differ diff --git a/crowdsec-docs/static/img/console/security_engines/details-page.png b/crowdsec-docs/static/img/console/security_engines/details-page.png deleted file mode 100644 index 86c71c1d..00000000 Binary files a/crowdsec-docs/static/img/console/security_engines/details-page.png and /dev/null differ diff --git a/crowdsec-docs/static/img/console_integrations_dark.png b/crowdsec-docs/static/img/console_integrations_dark.png index 184bd046..e8f592d7 100644 Binary files a/crowdsec-docs/static/img/console_integrations_dark.png and b/crowdsec-docs/static/img/console_integrations_dark.png differ diff --git a/crowdsec-docs/static/img/console_integrations_light.png b/crowdsec-docs/static/img/console_integrations_light.png index 55454b84..ff840bb6 100644 Binary files a/crowdsec-docs/static/img/console_integrations_light.png and b/crowdsec-docs/static/img/console_integrations_light.png differ diff --git a/crowdsec-docs/static/img/console_integrations_sophos_card_dark.png b/crowdsec-docs/static/img/console_integrations_sophos_card_dark.png new file mode 100644 index 00000000..3dcc2792 Binary files /dev/null and b/crowdsec-docs/static/img/console_integrations_sophos_card_dark.png differ diff --git a/crowdsec-docs/static/img/console_integrations_sophos_card_light.png b/crowdsec-docs/static/img/console_integrations_sophos_card_light.png new file mode 100644 index 00000000..1e09e899 Binary files /dev/null and b/crowdsec-docs/static/img/console_integrations_sophos_card_light.png differ diff --git a/crowdsec-docs/static/img/console_integrations_sophos_credentials_dark.png b/crowdsec-docs/static/img/console_integrations_sophos_credentials_dark.png new file mode 100644 index 00000000..390d85ef Binary files /dev/null and b/crowdsec-docs/static/img/console_integrations_sophos_credentials_dark.png differ diff --git a/crowdsec-docs/static/img/console_integrations_sophos_credentials_light.png b/crowdsec-docs/static/img/console_integrations_sophos_credentials_light.png new file mode 100644 index 00000000..7589cb4d Binary files /dev/null and b/crowdsec-docs/static/img/console_integrations_sophos_credentials_light.png differ diff --git a/crowdsec-docs/static/img/crowdsec_og_image.png b/crowdsec-docs/static/img/crowdsec_og_image.png new file mode 100644 index 00000000..3f638420 Binary files /dev/null and b/crowdsec-docs/static/img/crowdsec_og_image.png differ diff --git a/crowdsec-docs/static/img/details-page-remediation.png b/crowdsec-docs/static/img/details-page-remediation.png new file mode 100644 index 00000000..2231ee39 Binary files /dev/null and b/crowdsec-docs/static/img/details-page-remediation.png differ diff --git a/crowdsec-docs/static/img/firewall-bouncer-metrics.png b/crowdsec-docs/static/img/firewall-bouncer-metrics.png new file mode 100644 index 00000000..6325a274 Binary files /dev/null and b/crowdsec-docs/static/img/firewall-bouncer-metrics.png differ diff --git a/crowdsec-docs/static/img/usage_metrics_bouncer_os.png b/crowdsec-docs/static/img/usage_metrics_bouncer_os.png new file mode 100644 index 00000000..1fe1dec8 Binary files /dev/null and b/crowdsec-docs/static/img/usage_metrics_bouncer_os.png differ diff --git a/crowdsec-docs/static/img/usage_metrics_cscli_example.png b/crowdsec-docs/static/img/usage_metrics_cscli_example.png new file mode 100644 index 00000000..fb670343 Binary files /dev/null and b/crowdsec-docs/static/img/usage_metrics_cscli_example.png differ diff --git a/crowdsec-docs/static/img/usage_metrics_lp_cscli.png b/crowdsec-docs/static/img/usage_metrics_lp_cscli.png new file mode 100644 index 00000000..438c0111 Binary files /dev/null and b/crowdsec-docs/static/img/usage_metrics_lp_cscli.png differ diff --git a/crowdsec-docs/unversioned/bouncers/firewall.mdx b/crowdsec-docs/unversioned/bouncers/firewall.mdx index 7287ecff..1c41f20f 100644 --- a/crowdsec-docs/unversioned/bouncers/firewall.mdx +++ b/crowdsec-docs/unversioned/bouncers/firewall.mdx @@ -166,6 +166,35 @@ table ip6 crowdsec6 { ``` +## Metrics + +:::info +CrowdSec v1.6.3 and Firewall Remediation Component v0.0.30 are minimum versions required to have metrics. +::: + +You can check the metrics generated by the firewall-bouncer using the command `cscli metrics show bouncers`. + +![firewall-bouncer-metrics](/img/firewall-bouncer-metrics.png) + +Each line in the output represents a different source of blocked IPs, along with detailed metrics. + +- `Origin` refers to the name of the source, which could be: + - `CAPI` - The community blocklist that you receive in exchange for the information you provide to the network + - `crowdsec (security engine)` - The decisions made by your Security Engine based on triggered scenarios + - `lists:*` - Various lists to which you are subscribed +- `active_decisions IPs` represents the number of IPs contained in the respective list +- `dropped bytes & packets` indicates the number of bytes and packets dropped by the firewall due to the actions of the specified origin +- `processed bytes & packets` is only present for the `Total` line, as it denotes the overall number of bytes and packets processed by your firewall. + + +As the firewall bouncer operates at the network level, most malicious programs will not progress beyond attempting to establish a connection (and being denied). Therefore, metrics cannot reflect the "potentially saved traffic." + +### Ipset only mode + +If you are running ipset only mode, crowdsec-firewall-bouncer tries parsing the output to produce metrics, but: + - "managed" firewalls such ufw might confuse parser and lead to inconsistent metrics. + - "total" counters amount since the machine start, or iptables counter are reset, which can lead to inconsistent metrics. + ## Configuration Reference You can find a default configuration hosted on the [Github Repository](https://github.com/crowdsecurity/cs-firewall-bouncer/blob/main/config/crowdsec-firewall-bouncer.yaml) this is provided with the installation package. diff --git a/crowdsec-docs/unversioned/console/blocklists/integrations/firewall.md b/crowdsec-docs/unversioned/console/blocklists/integrations/firewall.md index 9af0bae9..4a46c396 100644 --- a/crowdsec-docs/unversioned/console/blocklists/integrations/firewall.md +++ b/crowdsec-docs/unversioned/console/blocklists/integrations/firewall.md @@ -36,6 +36,7 @@ Every product product has its way to handle external blocklists. We provide a si | [F5](https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/big-ip-network-firewall-policies-and-implementations-14-0-0/07.html) | Custom | `192.168.38.187,32,BL,crowdsec-myf5Integration`
`192.168.38.188,32,BL,crowdsec-myf5Integration` | | [Fortinet](https://docs.fortinet.com/document/fortigate/6.4.5/administration-guide/891236/external-blocklist-policy) | Plain text | `192.168.38.187`
`192.168.38.186` | | [Palo Alto](https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/external-dynamic-list#idf36cb80a-77f1-4d17-9c4b-7efe9fe426af) | Plain text | `192.168.38.187`
`192.168.38.186` | +| [Sophos](https://docs.sophos.com/nsg/sophos-firewall/21.0/help/en-us/webhelp/onlinehelp/AdministratorHelp/ActiveThreatResponse/ThirdPartyThreatFeeds/index.html) | Plain text | `192.168.38.187`
`192.168.38.186` | | Generic vendor | Plain text | `192.168.38.187`
`192.168.38.186` | ## How to bypass provider limit? diff --git a/crowdsec-docs/unversioned/console/blocklists/subscription.md b/crowdsec-docs/unversioned/console/blocklists/subscription.md index 965c8576..c40b5e00 100644 --- a/crowdsec-docs/unversioned/console/blocklists/subscription.md +++ b/crowdsec-docs/unversioned/console/blocklists/subscription.md @@ -1,11 +1,25 @@ # Blocklist Subscription -To begin the subscription process, click the _Subscribe_ button at the top of the blocklist details page. This will bring you to the bottom of the page, where you can subscribe to either Security Engines or integrations. +To begin the subscription process, click the _Subscribe_ button at the top of the blocklist details page. This will open a popup that will allow you to select the subscription method. ![](/img/console/blocklists/header_point_subscribe.png) ## Subscribe +You can subscribe to a blocklist at the Organization, Security Engine, or Integration level. On the community plan you can subscribe to 3 blocklists, to remove this limitation you can upgrade to the [enterprise plan](https://www.crowdsec.net/pricing) which includes various perks. + +### Organization Level + +The simplest way to subscribe to a blocklist is at the organization level. This will apply the blocklist to all Security Engines and Integrations within the organization this will include current and future Security Engines and Integrations. + +![](/img/console/blocklists/org_subscribe_popup.png) + +:::note +One remediation type will be applied to all the Security Engines and Integrations subscribed to the blocklist. +::: + +If you want to apply different remediation methods to different Security Engines or Integrations, you will need to subscribe to the blocklist at the Security Engine or Integration level. + ### Security Engines If your account already includes enrolled Security Engines, you'll find a section at the bottom that can also be used to start the subscription process. @@ -54,11 +68,11 @@ When performing an action, a popup will prompt to validate the action performed. ### Integrations -If your organization already has integrations, a section at the bottom can also be used to start the subscription process. +If your organization already has Integrations, a section at the bottom can also be used to start the subscription process. ![](/img/console/blocklists/integrations/subscriptions.png) To subscribe to an integration to the current blocklist, click on the _Subscribe_ button from the desired integration. -You will now see a different border around the subscribed integrations. +You will now see a different border around the subscribed Integrations. ![](/img/console/blocklists/integrations/subscribed.png) diff --git a/crowdsec-docs/unversioned/console/security_engines/details_page.md b/crowdsec-docs/unversioned/console/security_engines/details_page.md index 48e786c4..444da488 100644 --- a/crowdsec-docs/unversioned/console/security_engines/details_page.md +++ b/crowdsec-docs/unversioned/console/security_engines/details_page.md @@ -7,7 +7,7 @@ description: Learn how to view the details of a Security Engine in the CrowdSec This page will reference information about a specific Security Engine. This page is your one-stop resource for understanding everything related to the Security Engine you're interested in. -![Security Engine details page](/img/console/security_engines/details-page.png) +![Security Engine details page](/img/console/security_engines/details-page.jpeg) ## Usage @@ -19,21 +19,35 @@ At the top of the page, the essential information regarding the Security Engine Quick actions are available from the summary to apply changes to your Security Engine. -- [Update name or tags](/console/security_engines/name_and_tags.md) -- [Transfer an Engine](/console/security_engines/transfer_engine.md) -- [Remove an Engine](/console/security_engines/remove_engine.md) +- [Update name or tags](/console/security_engines/name_and_tags.md) +- [Transfer an Engine](/console/security_engines/transfer_engine.md) +- [Remove an Engine](/console/security_engines/remove_engine.md) ![Security Engine details page](/img/console/security_engines/details-page-actions.png) -### Log Processors +### Remediation components -The Log Processors section will only be displayed if the Security Engines have multiple log processors, indicating a Distributed Setup. Here, you can access all essential information regarding the log processors and their current version. +The [remediation component](/bouncers/intro.md) in CrowdSec will apply either the decisions made by CrowdSec, the blocklists or the custom decisions. -:::info -A warning will be displayed if any Security Engine has an outdated version. -::: +![Security Engine details page](/img/console/security_engines/details-page-remediation.png) -![Security Engine details page](/img/console/security_engines/details-page-log-processors.png) +#### Metrics + +Starting from version 1.6.3, CrowdSec’s remediation components now display detailed metrics. These metrics provide valuable insights into the number of traffic drops and the volume of traffic processed by each remediation component. + +To access a detailed view of these metrics, simply click the **Get More Info** button on any active remediation component card. This will show you the effectiveness of each decision made by the Security Engine, based on the installed blocklists. + +![Security Engine details page](/img/console/security_engines/details-page-remediation-metrics.png) + +In the same modal, you can view the active decisions. This section provides information about the number of decisions made by each source of decisions. + +![Security Engine details page](/img/console/security_engines/details-page-remediation-decisions.png) + +#### Inactive remediation components + +Remediation components are meant to block attackers. Having inactive remediation component can compromise the security of your Security Engine, as they cannot apply decisions. + +![Security Engine details page](/img/console/security_engines/details-page-inactive-bouncer.png) ### Blocklists @@ -57,14 +71,12 @@ By clicking on a scenario, you can access essential information about the scenar ![Security Engine details page](/img/console/security_engines/details-page-scenarios-hub.png) -### Remediation components - -The [remediation component](/bouncers/intro.md) in CrowdSec will apply either the decisions made by CrowdSec or the custom decisions. The complete list of decisions from the dedicated section is available at the bottom of the page. - -![Security Engine details page](/img/console/security_engines/details-page-remediation.png) +### Log Processors -#### Inactive remediation components +The Log Processors section will only be displayed if the Security Engines have multiple log processors, indicating a Distributed Setup. Here, you can access all essential information regarding the log processors and their current version. -Remediation components are meant to block attackers. Having inactive remediation component can compromise the security of your Security Engine, as they cannot apply decisions. +:::info +A warning will be displayed if any Security Engine has an outdated version. +::: -![Security Engine details page](/img/console/security_engines/details-page-inactive-bouncer.png) +![Security Engine details page](/img/console/security_engines/details-page-log-processors.png) diff --git a/crowdsec-docs/unversioned/cti_api/taxonomy/false_positives.mdx b/crowdsec-docs/unversioned/cti_api/taxonomy/false_positives.mdx index 028cd5df..37a6102d 100644 --- a/crowdsec-docs/unversioned/cti_api/taxonomy/false_positives.mdx +++ b/crowdsec-docs/unversioned/cti_api/taxonomy/false_positives.mdx @@ -4,10 +4,11 @@ title: False Positives sidebar_position: 7 --- -import TableRender from '@site/src/components/tableRender'; -import GithubIconRender from '@site/src/components/githubIconRender'; +import TableRender from "@site/src/components/tableRender" +import GithubIconRender from "@site/src/components/githubIconRender" -export const fpURL = "https://hub-cdn.crowdsec.net/master/taxonomy/false_positives.json"; +export const fpURL = + "https://hub-cdn.crowdsec.net/master/taxonomy/false_positives.json" export const columns = [ { header: "Name", @@ -17,10 +18,33 @@ export const columns = [ header: "Description", accessorKey: "description", }, -]; +] - +## How to Get Tagged as a False Positive + +To be able to be classified as a false positive, you need a proper technical justification of why your IP might be misclassified as a threat. This part is to be reviewed and validated by crowdsec. + +You also need public documentation stating the IP, ranges, and/or reverse DNS associated with the assets in question. This data must be machine-readable (no HTML, no PDF, etc.). + +Once your IP addresses are publicly available and accessible via HTTPS, you can contact support@crowdsec.net. Please include the URL of your IPs and ranges. + +The CrowdSec team will do their best to update the CTI with false positive information, so your IPs are flagged correctly. + +Here are some examples of providers who share their IPs and ranges: + +- [Bing](https://www.bing.com/toolbox/bingbot.json) +- [Google Bot](https://developers.google.com/search/apis/ipranges/googlebot.json) +- [Cloudfront](https://d7uri8nf7uskq.cloudfront.net/tools/list-cloudfront-ips) +- [Fastly](https://api.fastly.com/public-ip-list) + +:::note + +You don’t need to follow a specific format for the exposed list, but it’s recommended to keep the same format over time. Otherwise, the false positive enrichment may stop working. + +It’s best to use CSV or JSON for the list format. + +::: diff --git a/crowdsec-docs/unversioned/getting_started/next_steps.md b/crowdsec-docs/unversioned/getting_started/next_steps.md index 44e597a2..d928618b 100644 --- a/crowdsec-docs/unversioned/getting_started/next_steps.md +++ b/crowdsec-docs/unversioned/getting_started/next_steps.md @@ -11,7 +11,18 @@ The CrowdSec Console is a web-based interface provided by CrowdSec, offering a w See the dedicated [CrowdSec Console](/getting_started/post_installation/console.mdx) guide for more information. -### 2. Acquisitions + +### 2. Whitelists + +:::info +Whitelists are a way to tell CrowdSec to ignore certain events or IP addresses. +::: + +By default CrowdSec will whitelist private LAN IP addresses, however you may want to whitelist additional IP addresses or events. + +See the dedicated [Whitelists](/getting_started/post_installation/whitelists.mdx) guide for more information. + +### 3. Acquisitions :::info Acquisitions are sources of logs that CrowdSec can analyze. @@ -21,7 +32,7 @@ By default when CrowdSec is installed it will attempt to detect the running serv See the dedicated [Acquisition](/getting_started/post_installation/acquisition.mdx) guide for more information. -### 3. Profiles +### 4. Profiles :::info Profiles are a set of rules that drives what decisions will be taken by CrowdSec. @@ -31,7 +42,7 @@ CrowdSec comes with a default profile that is suitable for most use cases. Howev See the dedicated [Profiles](/getting_started/post_installation/profiles.mdx) guide for more information. -### 4. Metrics +### 5. Metrics :::info Metrics are a way to monitor the behavior of CrowdSec. diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/whitelists.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/whitelists.mdx new file mode 100644 index 00000000..18704154 --- /dev/null +++ b/crowdsec-docs/unversioned/getting_started/post_installation/whitelists.mdx @@ -0,0 +1,227 @@ +--- +id: whitelists +title: Whitelists +pagination_prev: getting_started/next_steps +pagination_next: getting_started/next_steps +--- + +# Whitelists + +Whitelists are a way to tell CrowdSec to ignore certain events or IP addresses. This can be useful if you have a static IP address that you know is safe, or if you have a service that _could_ generates a lot of false triggers by loading alot of thumbnails, images or fonts. + +By default CrowdSec will whitelist private LAN IP addresses via [this parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/whitelists), however you may want to whitelist additional IP addresses or events. + +In this guide we will cover how to create some whitelists and how to apply them to your CrowdSec instance. + +However, we can't cover every possible use case, so if you have a specific one that isn't covered here, please refer to [detailed whitelist documentation](/docs/next/whitelist/intro) + +:::info +We use "event" as a term to describe a log line that is currently being processed by CrowdSec. An "event" can be a log line from a log file, or an "overflow" from a scenario. +::: + +## Whitelist Types + +There are two different types of whitelists in CrowdSec: +- Parser (at enrich stage) +- Postoverflow + +Its important to note where these whitelists are applied in the pipeline, as this will affect the behavior and the context of the whitelists. + +### Parser + +Parser based whitelists are applied whilst the event is being Enriched. This means that the whitelists will prevent the event (log line) from hitting the scenario stage, leading to better memory management. + +typically these files are located in the following locations depending on your OS: +- Linux: `/etc/crowdsec/parsers/s02-enrich/` +- Freebsd: `/usr/local/etc/crowdsec/parsers/s02-enrich/` +- Windows: `c:/programdata/crowdsec/config/parsers/s02-enrich/` + +### Postoverflow + +Postoverflow whitelists are applied after a scenario has been triggered. You may want use this stage to do _"expensive"_ whitelists, such as DNS lookups. + +:::info +We deem a whitelist as _"expensive"_ if it requires a network call or a lookup that could potentially slow down the processing of the event. +::: + +typically these files are located in the following locations depending on your OS: +- Linux: `/etc/crowdsec/postoverflows/s01-whitelist/` +- Freebsd: `/usr/local/etc/crowdsec/postoverflows/s01-whitelist/` +- Windows: `c:/programdata/crowdsec/config/postoverflows/s01-whitelist/` + +*Postoverflow whitelist folders do not exist by default so you **MUST** manually create them* + +### Which one should I use? + +If you know which IP or event pattern you want to whitelist (for example a URI), then you should use a Parser whitelist. If you want to do a more complex whitelist, such as a DNS/rDNS lookup, then you should use the Postoverflow Whitelist. + +In short, enricher whitelists are applied to **every** event (log line), whereas postoverflow whitelists are only applied to **triggered** scenarios. + +## Should I create a whitelist? + +Whilst configuring CrowdSec for the first time you may want to create a whitelist for the following reasons: +- You have a static IP address that you want to ensure is never banned. +- You have a service that could generate a lot of false triggers. + +:::info +An example of a service that could generate a lot of false triggers is a web application that loads a lot of resources such a thumbnails, images, or fonts. +::: + +If you are unsure if you need a whitelist, you can monitor the logs and see if there are any false positives that you want to prevent. + +However, please note if you should get a decision made agaisnt your IP address and using remediation components such as `iptables` or `nftables` then this will prevent you from accessing **ANY** service on the server. + +Within our enterprise offering we have a feature called [Decision Management](console/decisions/decisions_management.md) that allows you to manage decisions via the [console](https://app.crowdsec.net/). + +## Creating your first whitelist. + +Depending on your criteria, you may want to create a whitelist for a specific IP address, a range of IP addresses, or a specific event pattern. + +We will provide examples for each of these scenarios. + +:::note +The example location shown is for Linux, you will need to adjust the path based on your OS as noted above. +::: + +### Static IP address + +If you want to whitelist a specific IP address for example `192.168.1.1`, you can create a file in the Enricher folder with the following content: + +```yaml title="/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml" +name: my/whitelist ## Must be unqiue +description: "Whitelist events from my IP" +whitelist: + reason: "My IP" + ip: + - "192.168.1.1" +``` + +Once you have created the file you will need to restart the CrowdSec service for the changes to take effect. + +```bash title="Restart CrowdSec" +sudo systemctl restart crowdsec +``` + +### IP Range + +If you want to whitelist a range of IP addresses, for example `192.168.1.0/24` you can create a file in the Enricher folder with the following content: + +```yaml title="/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml" +name: my/whitelist ## Must be unqiue +description: "Whitelist events from my IP range" +whitelist: + reason: "My IP range" + cidr: + - "192.168.1.0/24" +``` + +Once you have created the file you will need to restart the CrowdSec service for the changes to take effect. + +```bash title="Restart CrowdSec" +sudo systemctl restart crowdsec +``` + +### Expression pattern + +If you want to whitelist a specific event pattern, for example http log line that is a healthcheck so typically a `GET` request to `/health` you can create a file in the Enricher folder with the following content: + +```yaml title="/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml" +name: my/whitelist ## Must be unqiue +description: "Whitelist events with GET /health" +filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']" +whitelist: + reason: "GET /health" + expression: + - "evt.Meta.http_verb == 'GET' && evt.Meta.http_path == '/health'" +``` + +:::note +This will discard any event that has a `http_verb` of `GET` and a `http_path` of `/health` no matter the origin. +::: + +Once you have created the file you will need to restart the CrowdSec service for the changes to take effect. + +```bash title="Restart CrowdSec" +sudo systemctl restart crowdsec +``` + +Expression whitelists are very powerful and can be used to whitelist based on any field in the event. You can find a more detailed version of the [expression guide here](/docs/next/whitelist/create_expr) which will showcase how you can find which fields are available to you based on the log line you are processing. + +### Dynamic IP address + +If you want to whitelist an IP address that is not static, you need to use a external DDNS service to resolve the IP address and then use the Postoverflow whitelist to whitelist the resolved IP address. + +:::note +This is a postoverflow whitelist as it requires a network call to resolve the IP address. +::: + +```yaml title="/etc/crowdsec/postoverflows/s01-whitelist/01-my-whitelist.yaml" +name: my/whitelist ## Must be unqiue +description: "Whitelist events from my dynamic IP" +whitelist: + reason: "My dynamic IP" + expression: + - evt.Overflow.Alert.Source.IP in LookupHost("foo.com") +``` + +:::warning +Please read the [LookupHost](/docs/next/expr/ip_helpers/#lookuphosthost-string-string) function documentation before altering the current example. +::: + +## Testing your whitelist + +Once you have created your whitelist, you can test it by running a log line through the parser and checking the output. + +:::info +In the example we using Nginx logs, you will need to adjust the testing to point towards your log file. +::: + +```bash title="Test an IP/Range whitelist" +grep 192.168.1.1 /var/log/nginx/access.log | tail -n 1 | sudo cscli explain -f- --type nginx +``` + +```bash title="Test an Expression whitelist" +grep /health /var/log/nginx/access.log | tail -n 1 | sudo cscli explain -f- --type nginx +``` + +```bash title="Example Output" +line: 192.168.1.1 - - [05/Sep/2024:18:07:25 +0000] "GET /health? HTTP/2.0" 200 42 "-" "curl/7.68.0" +├ s00-raw +| └ 🟢 crowdsecurity/non-syslog (+5 ~8) +├ s01-parse +| └ 🟢 crowdsecurity/nginx-logs (+23 ~2) +├ s02-enrich +| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2) +| ├ 🟢 crowdsecurity/geoip-enrich (+13) +| ├ 🟢 crowdsecurity/http-logs (+8 ~1) +#highlight-next-line +| └ 🟢 my/whitelist (~2 [whitelisted]) +#highlight-next-line +└-------- parser success, ignored by whitelist () 🟢 +``` + +:::info +Currently postoverflows are not supported by `cscli explain` so you will need to check the logs to see if the whitelist is working. +::: + +Example logs line you will see: + +```bash title="Example logs" +time="07-07-2020 17:11:09" level=info msg="Ban for 192.168.1.1 whitelisted, reason [My dynamic IP]" id=cold-sunset name=my/whitelist stage=s01 +``` + +## Whitelisted but there still a decision? + +Whitelisting an IP address or event will prevent the events from triggering **new** decisions, however, any existing decisions will still be applied. + +You must manually remove the decisions by running: + +```bash title="Remove decisions" +sudo cscli decisions delete --ip 192.168.1.1 +``` + +## Next Steps? + +If you are still looking for examples on how to create a whitelist, you can find more detailed documentation [here](/docs/next/whitelist/intro). + +If you have created your first whitelist, tested it and happy to continue then [click here](getting_started/next_steps.md#2-whitelists). diff --git a/crowdsec-docs/unversioned/integrations/intro.mdx b/crowdsec-docs/unversioned/integrations/intro.mdx index bfd15aef..5ea0f5bc 100644 --- a/crowdsec-docs/unversioned/integrations/intro.mdx +++ b/crowdsec-docs/unversioned/integrations/intro.mdx @@ -55,6 +55,7 @@ Once you are on the Integrations page you can select the integration you would l - [F5](integrations/f5.mdx) - [Fortinet](integrations/fortinet.mdx) - [Palo Alto](integrations/paloalto.mdx) +- [Sophos](integrations/sophos.mdx) - [Generic Firewall](integrations/genericvendor.mdx) - [Remediation Component](integrations/remediationcomponent.mdx) diff --git a/crowdsec-docs/unversioned/integrations/sophos.mdx b/crowdsec-docs/unversioned/integrations/sophos.mdx new file mode 100644 index 00000000..28e42c51 --- /dev/null +++ b/crowdsec-docs/unversioned/integrations/sophos.mdx @@ -0,0 +1,68 @@ +--- +id: sophos +title: Sophos +--- + +import ThemedImage from "@theme/ThemedImage"; +import useBaseUrl from "@docusaurus/useBaseUrl"; + +The CrowdSec Sophos integration allows you to block malicious IPs in your Sophos firewall. This guide will walk you through the steps to integrate CrowdSec blocklists with your Sophos firewall. + +### Prerequisites + +Before you begin, please ensure your Sophos device supports ingesting blocklists. If you are unsure, please refer to the Sophos documentation or contact Sophos support. + +### Steps + +We will presume you followed the [Getting Started](integrations/intro.mdx) guide and have created an account on the CrowdSec Console. + +Once you are authenticated, you can proceed to the Blocklist tab located on the top menu bar, from there you can select the Integrations sub menu. + +Once the page has loaded, you can click the "Connect" button under the Sophos logo. + + + +Doing so will prompt you to name this integration, you can name it anything you like, for example "My Integration ". Note the name should be unique per integration that is tied to your account. + + + +Once the integration is generated you will be presented with a credentials screen that will provide you with the necessary information to configure your Sophos Firewall. This information will **ONLY** be displayed once, so please ensure you copy it down. + + + +[Sophos Documentation](https://docs.sophosnetworks.com/pan-os/11-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/external-dynamic-list#idf36cb80a-77f1-4d17-9c4b-7efe9fe426af) + +## Format example + +The CrowdSec blocklist will be in plain text format, with one IP address per line. Here is an example of how the blocklist will look: + +``` +192.168.38.187 +192.168.38.186 +``` + +## Contribute to this documentation + +Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) + +## Next Steps + +Now that you have integrated CrowdSec integration with your Sophos Firewall, you can proceed to the [Blocklist Catalog](console/blocklists/catalog.md) to find what blocklists you can subscribe too. diff --git a/crowdsec-docs/unversioned/service_api/integrations.md b/crowdsec-docs/unversioned/service_api/integrations.md index 305ba025..62dc8e7e 100644 --- a/crowdsec-docs/unversioned/service_api/integrations.md +++ b/crowdsec-docs/unversioned/service_api/integrations.md @@ -9,11 +9,12 @@ For some constructors, the integrations can generate vendor-specific format, see | Constructor | Authentication | Multiple URLs | Constructor Doc | Format | | ----------- | -------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | -| Fortinet | Basic Auth | Yes | [Fortinet doc](https://docs.fortinet.com/document/fortigate/6.4.5/administration-guide/891236/external-blocklist-policy) | `fortigate` | -| Palo alto | Basic Auth | Yes | [PaloAlto doc](https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/configure-the-firewall-to-access-an-external-dynamic-list) | `paloalto` | | CheckPoint | Basic Auth | Yes | [CheckPoint doc](https://support.checkpoint.com/results/sk/sk1) | `checkpoint` | | Cisco | Basic Auth | Yes | [Cisco doc](https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/710/management-center-device-config-71/objects-object-mgmt.html#ID-2243-00000291) | `cisco` | | F5 | Basic Auth | Yes | [F5 doc](https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/big-ip-network-firewall-policies-and-implementations-14-0-0/07.html ) | `f5` | +| Fortinet | Basic Auth | Yes | [Fortinet doc](https://docs.fortinet.com/document/fortigate/6.4.5/administration-guide/891236/external-blocklist-policy) | `fortigate` | +| Palo alto | Basic Auth | Yes | [PaloAlto doc](https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/configure-the-firewall-to-access-an-external-dynamic-list) | `paloalto` | +| Sophos | Basic Auth | Yes | [Sophos doc](https://docs.sophos.com/nsg/sophos-firewall/21.0/help/en-us/webhelp/onlinehelp/AdministratorHelp/ActiveThreatResponse/ThirdPartyThreatFeeds/index.html) | `sophos` | For all the other providers, the `plain_text` format consists of one ip per line, and should be supported by most devices. If a specific format is missing, reach out to us and we'll help you support it! diff --git a/crowdsec-docs/unversioned/troubleshooting/cti.mdx b/crowdsec-docs/unversioned/troubleshooting/cti.mdx new file mode 100644 index 00000000..0615dc4b --- /dev/null +++ b/crowdsec-docs/unversioned/troubleshooting/cti.mdx @@ -0,0 +1,38 @@ +--- +title: Troubleshooting CTI +id: cti +--- + +## Community support + +Please try to resolve your issue by reading [the documentation](../cti_api/intro). If you're unable to find a solution, don't hesitate to seek assistance in: + +- [Discourse](https://discourse.crowdsec.net/) +- [Discord](https://discord.gg/crowdsec) + +## False Positive + +### How to Get Tagged as a False Positive + +To be able to be classified as a false positive, you need a proper technical justification of why your IP might be misclassified as a threat. This part is to be reviewed and validated by crowdsec. + +You also need public documentation stating the IP, ranges, and/or reverse DNS associated with the assets in question. This data must be machine-readable (no HTML, no PDF, etc.). + +Once your IP addresses are publicly available and accessible via HTTPS, you can contact support@crowdsec.net. Please include the URL of your IPs and ranges. + +The CrowdSec team will do their best to update the CTI with false positive information, so your IPs are flagged correctly. + +Here are some examples of providers who share their IPs and ranges: + +- [Bing](https://www.bing.com/toolbox/bingbot.json) +- [Google Bot](https://developers.google.com/search/apis/ipranges/googlebot.json) +- [Cloudfront](https://d7uri8nf7uskq.cloudfront.net/tools/list-cloudfront-ips) +- [Fastly](https://api.fastly.com/public-ip-list) + +:::note + +You don’t need to follow a specific format for the exposed list, but it’s recommended to keep the same format over time. Otherwise, the false positive enrichment may stop working. + +It’s best to use CSV or JSON for the list format. + +::: diff --git a/crowdsec-docs/unversioned/troubleshooting/intro.md b/crowdsec-docs/unversioned/troubleshooting/intro.md index e068d45c..c103d4b8 100644 --- a/crowdsec-docs/unversioned/troubleshooting/intro.md +++ b/crowdsec-docs/unversioned/troubleshooting/intro.md @@ -15,12 +15,14 @@ We have extended our troubleshooting documentation to cover more common issues a ### [Remediation Components](/troubleshooting/remediation_components.mdx) +### [CTI](/troubleshooting/cti.mdx) + ## Community support Please try to resolve your issue by reading the documentation. If you're unable to find a solution, don't hesitate to seek assistance in: -- [Discourse](https://discourse.crowdsec.net/) -- [Discord](https://discord.gg/crowdsec) +- [Discourse](https://discourse.crowdsec.net/) +- [Discord](https://discord.gg/crowdsec) # FAQ @@ -64,9 +66,9 @@ If you need help for large scale deployment, please get in touch with us on the Setting up a proxy works out of the box, the [net/http golang library](https://golang.org/src/net/http/transport.go) can handle those environment variables: -* `HTTP_PROXY` -* `HTTPS_PROXY` -* `NO_PROXY` +- `HTTP_PROXY` +- `HTTPS_PROXY` +- `NO_PROXY` For example: @@ -75,6 +77,7 @@ export HTTP_PROXY=http://: ``` #### Systemd variable + On Systemd devices you have to set the proxy variable in the environment section for the CrowdSec service. To avoid overwriting the service file during an update, a folder is created in `/etc/systemd/system/crowdsec.service.d` and a file in it named `http-proxy.conf`. The content for this file should look something like this: ```bash title="systemctl edit crowdsec.service" @@ -90,6 +93,7 @@ Then you can restart CrowdSec like this: `systemctl restart crowdsec` #### Sudo + If you use `sudo cscli`, just add this line in `visudo` after setting up the previous environment variables: ``` @@ -146,20 +150,22 @@ CrowdSec Hub should be used when you have an issue with a parser, scenario or co To disable the central API, simply comment out the [`online_client` section of the configuration file](/docs/next/configuration/crowdsec_configuration#online_client). -### Why are some scenarios/parsers "tainted" or "custom" ? +### Why are some scenarios/parsers "tainted" or "custom" ? When using `cscli` to list your parsers, scenarios and collections, some might appear as "tainted" or "local". "tainted" items: - - Originate from the hub - - Were locally modified - - Will not be automatically updated/upgraded by `cscli` operations (unless `--force` or similar is specified) - - Won't be sent to Central API and won't appear in the Console (unless `cscli console enable tainted` has been specified) + +- Originate from the hub +- Were locally modified +- Will not be automatically updated/upgraded by `cscli` operations (unless `--force` or similar is specified) +- Won't be sent to Central API and won't appear in the Console (unless `cscli console enable tainted` has been specified) "local" items: - - Have been locally created by the user - - Are not managed by `cscli` operations - - Won't be sent to Central API and won't appear in the Console (unless `cscli console enable custom` has been specified) + +- Have been locally created by the user +- Are not managed by `cscli` operations +- Won't be sent to Central API and won't appear in the Console (unless `cscli console enable custom` has been specified) ### Which information is sent to your services ? @@ -201,6 +207,7 @@ line: May 16 07:50:30 sd-126005 sshd[10041]: Invalid user git from 78.142.18.204 ├ 🟢 crowdsecurity/ssh-slow-bf └ 🟢 crowdsecurity/ssh-slow-bf_user-enum ``` + This command will allow you to see each parser behavior. :::warning diff --git a/crowdsec-docs/unversioned/troubleshooting/security_engine.mdx b/crowdsec-docs/unversioned/troubleshooting/security_engine.mdx index 8306531a..750514cb 100644 --- a/crowdsec-docs/unversioned/troubleshooting/security_engine.mdx +++ b/crowdsec-docs/unversioned/troubleshooting/security_engine.mdx @@ -202,6 +202,35 @@ You can use the os related commands to filter the logs to only show errors. If it's a configuration file issue, the error message may log in a different location. As an example, if a parser/scenario file is invalid, the error message will be logged within the configured log file, however, if `config.yaml` is invalid, the error message will be logged in syslog instead. ::: +### Hash Mismatch + - **error** message might look like: +``` +level=fatal msg="error while installing 'crowdsecurity/traefik': while downloading crowdsecurity/http-cve: while downloading crowdsecurity/http-cve to https://hub-cdn.crowdsec.net/master/collections/crowdsecurity/http-cve.yaml: hash mismatch: expected dd2e8debbba19d19646b9a8010baf9bd901a4eec84a53a0cb62294b8da3e91ef, got b974f20dc2cf23c292eede422c1c6e0008297f8b3397e3851bb6a988866f3e34" +``` +A hash mismatch may occur when the local `.index.json` (hub index) is out of date with the current latest. + +Solutions differ based on how you are running CrowdSec: + - **Bare Metal Install**: Run `cscli hub update` on the machine and this will update the index file for you to run the previous install command. + - **Docker/Podman**: The docker startup script will automatically try to update the hub index if there is an update pending. + +:::info +Within the container the hub index will **NOT** be automatically updated if the modification time of the file is less than `24 hours` from when the container starts so to force a index update you can manually run a temporary container +::: + +#### Temporary container to update index + +```bash +docker run --rm -it --entrypoint cscli -v $MountPath:/etc/crowdsec/ -v $MountPath:/var/lib/crowdsec/data/ crowdsecurity/crowdsec:latest hub update +``` + +In the example above I have used `$MountPath` as place holders for you to update where the mounts are if they are relative via your compose then launch the run within the compose directory with the values updated, if they are named volumes then replace with the named volumes. The names are volumes may have a prefix so check via `docker volume ls` for the actual names. + +Once the temporary container has run and terminated you will be able to bring up the container as normal via `docker compose up -d` + +:::note +If you are running a previous version of CrowdSec then please update the `docker run` command to us the same tag as the one deployed +::: + ### Cannot bind to the configured port or IP - **error** message might look like: ``` diff --git a/crowdsec-docs/unversioned/user_guides/machines_management.mdx b/crowdsec-docs/unversioned/user_guides/machines_management.mdx index 30c457fb..9e17f7bf 100644 --- a/crowdsec-docs/unversioned/user_guides/machines_management.mdx +++ b/crowdsec-docs/unversioned/user_guides/machines_management.mdx @@ -101,3 +101,42 @@ import CodeBlock from '@theme/CodeBlock'; + +### Machine auto validation + +:::warning + +If you enabled this feature, make sure to restrict the IP ranges as much as possible. + +Any rogue machine registered in your LAPI will be able to push arbitrary alerts, and potentially lock you out. + +::: + +In some situation, it's not practical to manually create or validate new machines in LAPI (eg, when running in an environment that uses auto-scaling). + +It is possible to configure LAPI to automatically accept new machines upon creation with the `api.server.auto_registration` section: + +```yaml + api: + server: + auto_registration: + enabled: true + token: "long_token_that_is_at_least_32_characters_long" + allowed_ranges: + - 10.2.0.42/32 + - 10.0.0.0/24 +``` + +You have to specify both the `token` and `allowed_ranges`. + +Once the configuration is done, you can pass the token to your registration request with the `--token` parameter: + +```bash +cscli lapi register --machine my_machine --token long_token_that_is_at_least_32_characters_long +``` + +If the token is valid and the request is coming from an authorized IP range, LAPI will automatically validate the machine and it will be able to login without any further configuration. + +If no token is sent, LAPI will treat the request as a normal registration, regardless of the configuration. + +If a token is set but invalid, the request will be refused. \ No newline at end of file