Skip to content

Commit

Permalink
Merge pull request #744 from ferozsalam/bump-elasticsearch-py
Browse files Browse the repository at this point in the history
Add Elasticsearch 8 support + remove ES 6 support
  • Loading branch information
jertel authored Mar 5, 2022
2 parents 7d80a6b + e69829e commit 61777fa
Show file tree
Hide file tree
Showing 42 changed files with 283 additions and 1,846 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,16 @@
# 2.x.x

## Breaking changes
- Add support for Elasticsearch 8, remove support for Elasticsearch 6 and below - [#744](https://github.com/jertel/elastalert2/pull/744) - @ferozsalam, @jertel, and @nsano-rururu
WARNING! Read the [ES 8 upgrade notes](https://elastalert2.readthedocs.io/en/latest/recipes/faq.html#does-elastalert-2-support-elasticsearch-8) BEFORE upgrading your cluster to Elasticsearch 8. Failure to do so can result in your cluster no longer starting and unable to rollback to 7.x.
- Kibana dashboard integration has been removed, as it only was supported with older versions of Elasticsearch and Kibana. Per the above breaking change those older versions are no longer supported by ElastAlert 2.
- Dockerfile refactor for app home and user home to be the same directory (/opt/elastalert/). Before app home is /opt/elastalert/ and user home is /opt/elastalert/elastalert. After app home and user home are the same /opt/elastalert/ - [#656](https://github.com/jertel/elastalert2/pull/656)

## New features
- [MS Teams] Kibana Discover URL and Facts - [#660](https://github.com/jertel/elastalert2/pull/660) - @thib12
- Add support for Kibana 7.17 for Kibana Discover - [#695](https://github.com/jertel/elastalert2/pull/695) - @nsano-rururu
- Added a fixed name metric_agg_value to MetricAggregationRule match_body - [#697](https://github.com/jertel/elastalert2/pull/697) - @iamxeph

## Other changes
- Load Jinja template when loading an alert - [#654](https://github.com/jertel/elastalert2/pull/654) - @thib12
- tox 3.24.4 to 3.24.5 - [#655](https://github.com/jertel/elastalert2/pull/655) - @nsano-rururu
Expand Down
1 change: 0 additions & 1 deletion chart/elastalert2/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,6 @@ extraConfigOptions: {}
# # Options to propagate to all rules, e.g. a common slack_webhook_url or kibana_url
# # Please note at the time of implementing this value, it will not work for required_locals
# # Which MUST be set at the rule level, these are: ['alert', 'type', 'name', 'index']
# generate_kibana_link: true
# kibana_url: https://kibana.yourdomain.com
# slack_webhook_url: dummy

Expand Down
2 changes: 1 addition & 1 deletion docs/source/elastalert.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Additional rule types and alerts can be easily imported or written. (See :ref:`W

In addition to this basic usage, there are many other features that make alerts more useful:

- Alerts link to Kibana dashboards
- Alerts link to Kibana Discover searches
- Aggregate counts for arbitrary fields
- Combine alerts into periodic reports
- Separate alerts by using a unique key field
Expand Down
18 changes: 8 additions & 10 deletions docs/source/recipes/faq-md.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,8 +219,7 @@ rounded to a single timestamp.

If you are using ``query_key`` (a single key, not multiple keys) you can use ``use_terms_query``.
This will make ElastAlert 2 perform a terms aggregation to get the counts for each value of a certain
field. Both ``use_terms_query`` and ``use_count_query`` also require ``doc_type`` to be set to the
``_type`` of the documents. They may not be compatible with all rule types.
field. May not be compatible with all rule types.

Can I perform aggregations?
==========
Expand Down Expand Up @@ -436,21 +435,20 @@ filter:
Does ElastAlert 2 support Elasticsearch 8?
===========

Support for Elasticsearch 8 is a work in progress. It is currently possible to
load ElastAlert 2 against a _fresh_ installation of Elasticsearch (i.e. one where
no previous ElastAlert instance has been running) without any extra steps.
ElastAlert 2 supports Elasticsearch 8.

To upgrade an existing ElastAlert 2 installation to Elasticsearch 8 the
following manual steps are required:
following manual steps are required (note the important WARNING below):

* Shutdown ElastAlert 2.
* Delete or rename the old `elastalert*` indices. See Elasticsearch
documentation for instructions on how to delete via the API.
* Delete the old `elastalert*` indices. See [Elasticsearch
documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html)
for instructions on how to delete via the API, or use the Kibana Index Management interface.
* Upgrade the Elastic cluster to Elasticsearch 8.
* If NOT running ElastAlert 2 via Docker or Kubernetes, run
elastalert-create-index to create the new indices. This is not needed when
running via a container since the container always attempts to creates the
indices at startup, if they're not yet created.
* Restart ElastAlert 2.

However, *at this point it is not guaranteed that features which used to work
on Elasticsearch 7 will still work*.
WARNING: Failure to remove the old ElastAlert indices can result in a non-working Elasticsearch cluster. This is because the ElastAlert indices contain deprecated features and the Elasticsearch 8 upgrade logic is currently flawed and does not correctly handle this situation. The Elasticsearch GitHub repository contains [more information](https://github.com/elastic/elasticsearch/issues/84199) on this problem.
2 changes: 1 addition & 1 deletion docs/source/recipes/writing_filters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ For ranges on fields::
Negation, and, or
*****************

For Elasticsearch 2.X, any of the filters can be embedded in ``not``, ``and``, and ``or``::
Any of the filters can be embedded in ``not``, ``and``, and ``or``::

filter:
- or:
Expand Down
93 changes: 11 additions & 82 deletions docs/source/ruletypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,22 +60,12 @@ Rule Configuration Cheat Sheet
+--------------------------------------------------------------+ |
| ``description`` (string, default empty string) | |
+--------------------------------------------------------------+ |
| ``generate_kibana_link`` (boolean, default False) | |
+--------------------------------------------------------------+ |
| ``use_kibana_dashboard`` (string, no default) | |
+--------------------------------------------------------------+ |
| ``kibana_url`` (string, default from es_host) | |
+--------------------------------------------------------------+ |
| ``kibana_username`` (string, no default) | |
+--------------------------------------------------------------+ |
| ``kibana_password`` (string, no default) | |
+--------------------------------------------------------------+ |
| ``use_kibana4_dashboard`` (string, no default) | |
+--------------------------------------------------------------+ |
| ``kibana4_start_timedelta`` (time, default: 10 min) | |
+--------------------------------------------------------------+ |
| ``kibana4_end_timedelta`` (time, default: 10 min) | |
+--------------------------------------------------------------+ |
| ``generate_kibana_discover_url`` (boolean, default False) | |
+--------------------------------------------------------------+ |
| ``shorten_kibana_discover_url`` (boolean, default False) | |
Expand Down Expand Up @@ -177,13 +167,9 @@ Rule Configuration Cheat Sheet
| ``attach_related`` (boolean, default False) | | | | | Opt | | | | | | | |
+-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
|``use_count_query`` (boolean, default False) | | | | | Opt | Opt | Opt | | | | | |
| | | | | | | | | | | | | |
|``doc_type`` (string, no default) | | | | | | | | | | | | |
+-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
|``use_terms_query`` (boolean, default False) | | | | | Opt | Opt | | Opt | | | | |
| | | | | | | | | | | | | |
|``doc_type`` (string, no default) | | | | | | | | | | | | |
| | | | | | | | | | | | | |
|``query_key`` (string or list, no default) | | | | | | | | | | | | |
| | | | | | | | | | | | | |
|``terms_size`` (int, default 50) | | | | | | | | | | | | |
Expand Down Expand Up @@ -224,8 +210,6 @@ Rule Configuration Cheat Sheet
| | | | | | | | | | | | | |
|([min|max|avg|sum|cardinality|value_count|percentiles])| | | | | | | | | | | | |
+-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
|``doc_type`` (string, no default) | | | | | | | | | | Req | Req | Req |
+-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
|``metric_agg_script`` (no default) | | | | | | | | | | Opt | Opt | |
+-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
|``percentile_range`` ++required if percentiles is used | | | | | | | | | | Req++ | Req++ | |
Expand Down Expand Up @@ -596,27 +580,13 @@ description
``description``: text describing the purpose of rule. (Optional, string, default empty string)
Can be referenced in custom alerters to provide context as to why a rule might trigger.

generate_kibana_link
^^^^^^^^^^^^^^^^^^^^

``generate_kibana_link``: This option is for Kibana 3 only.
If true, ElastAlert 2 will generate a temporary Kibana dashboard and include a link to it in alerts. The dashboard
consists of an events over time graph and a table with ``include`` fields selected in the table. If the rule uses ``query_key``, the
dashboard will also contain a filter for the ``query_key`` of the alert. The dashboard schema will
be uploaded to the kibana-int index as a temporary dashboard. (Optional, boolean, default False)

kibana_url
^^^^^^^^^^

``kibana_url``: The base url of the Kibana application. If not specified, a URL will be constructed using ``es_host``
and ``es_port``.

This value will be used if one of the following conditions are met:

- ``generate_kibana_link`` is true
- ``use_kibana_dashboard`` is true
- ``use_kibana4_dashboard`` is true
- ``generate_kibana_discover_url`` is true and ``kibana_discover_app_url`` is a relative path
This value will be used if ``generate_kibana_discover_url`` is true and ``kibana_discover_app_url`` is a relative path

(Optional, string, default ``http://<es_host>:<es_port>/_plugin/kibana/``)

Expand All @@ -636,38 +606,6 @@ This value is only used if ``shorten_kibana_discover_url`` is true.

(Optional, string, no default)

use_kibana_dashboard
^^^^^^^^^^^^^^^^^^^^

``use_kibana_dashboard``: The name of a Kibana 3 dashboard to link to. Instead of generating a dashboard from a template,
ElastAlert 2 can use an existing dashboard. It will set the time range on the dashboard to around the match time,
upload it as a temporary dashboard, add a filter to the ``query_key`` of the alert if applicable,
and put the url to the dashboard in the alert. (Optional, string, no default)

use_kibana4_dashboard
^^^^^^^^^^^^^^^^^^^^^

``use_kibana4_dashboard``: A link to a Kibana 4 dashboard. For example, "https://kibana.example.com/#/dashboard/My-Dashboard".
This will set the time setting on the dashboard from the match time minus the timeframe, to 10 minutes after the match time.
Note that this does not support filtering by ``query_key`` like Kibana 3. This value can use `$VAR` and `${VAR}` references
to expand environment variables.

kibana4_start_timedelta
^^^^^^^^^^^^^^^^^^^^^^^

``kibana4_start_timedelta``: Defaults to 10 minutes. This option allows you to specify the start time for the generated kibana4 dashboard.
This value is added in front of the event. For example,

``kibana4_start_timedelta: minutes: 2``

kibana4_end_timedelta
^^^^^^^^^^^^^^^^^^^^^

``kibana4_end_timedelta``: Defaults to 10 minutes. This option allows you to specify the end time for the generated kibana4 dashboard.
This value is added in back of the event. For example,

``kibana4_end_timedelta: minutes: 2``

generate_kibana_discover_url
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down Expand Up @@ -735,9 +673,8 @@ kibana_discover_version

The currently supported versions of Kibana Discover are:

- `5.6`
- `6.0`, `6.1`, `6.2`, `6.3`, `6.4`, `6.5`, `6.6`, `6.7`, `6.8`
- `7.0`, `7.1`, `7.2`, `7.3`, `7.4`, `7.5`, `7.6`, `7.7`, `7.8`, `7.9`, `7.10`, `7.11`, `7.12`, `7.13`, `7.14`, `7.15`, `7.16`, `7.17`
- `8.0`

``kibana_discover_version: '7.15'``

Expand All @@ -747,6 +684,8 @@ kibana_discover_index_pattern_id
``kibana_discover_index_pattern_id``: The id of the index pattern to link to in the Kibana Discover application.
These ids are usually generated and can be found in url of the index pattern management page, or by exporting its saved object.

In this documentation all references of "index pattern" refer to the similarly named concept in Kibana 8 called "data view".

Example export of an index pattern's saved object:

.. code-block:: text
Expand Down Expand Up @@ -1106,12 +1045,10 @@ Optional:

``use_count_query``: If true, ElastAlert 2 will poll Elasticsearch using the count api, and not download all of the matching documents. This is
useful is you care only about numbers and not the actual data. It should also be used if you expect a large number of query hits, in the order
of tens of thousands or more. ``doc_type`` must be set to use this.

``doc_type``: Specify the ``_type`` of document to search for. This must be present if ``use_count_query`` or ``use_terms_query`` is set.
of tens of thousands or more.

``use_terms_query``: If true, ElastAlert 2 will make an aggregation query against Elasticsearch to get counts of documents matching
each unique value of ``query_key``. This must be used with ``query_key`` and ``doc_type``. This will only return a maximum of ``terms_size``,
each unique value of ``query_key``. This must be used with ``query_key``. This will only return a maximum of ``terms_size``,
default 50, unique terms.

``terms_size``: When used with ``use_terms_query``, this is the maximum number of terms returned per query. Default is 50.
Expand Down Expand Up @@ -1241,12 +1178,10 @@ cause alerts. Baseline is established after ``timeframe`` has elapsed twice sinc

``use_count_query``: If true, ElastAlert 2 will poll Elasticsearch using the count api, and not download all of the matching documents. This is
useful is you care only about numbers and not the actual data. It should also be used if you expect a large number of query hits, in the order
of tens of thousands or more. ``doc_type`` must be set to use this.

``doc_type``: Specify the ``_type`` of document to search for. This must be present if ``use_count_query`` or ``use_terms_query`` is set.
of tens of thousands or more.

``use_terms_query``: If true, ElastAlert 2 will make an aggregation query against Elasticsearch to get counts of documents matching
each unique value of ``query_key``. This must be used with ``query_key`` and ``doc_type``. This will only return a maximum of ``terms_size``,
each unique value of ``query_key``. This must be used with ``query_key``. This will only return a maximum of ``terms_size``,
default 50, unique terms.

``terms_size``: When used with ``use_terms_query``, this is the maximum number of terms returned per query. Default is 50.
Expand All @@ -1268,12 +1203,10 @@ Optional:

``use_count_query``: If true, ElastAlert 2 will poll Elasticsearch using the count api, and not download all of the matching documents. This is
useful is you care only about numbers and not the actual data. It should also be used if you expect a large number of query hits, in the order
of tens of thousands or more. ``doc_type`` must be set to use this.

``doc_type``: Specify the ``_type`` of document to search for. This must be present if ``use_count_query`` or ``use_terms_query`` is set.
of tens of thousands or more.

``use_terms_query``: If true, ElastAlert 2 will make an aggregation query against Elasticsearch to get counts of documents matching
each unique value of ``query_key``. This must be used with ``query_key`` and ``doc_type``. This will only return a maximum of ``terms_size``,
each unique value of ``query_key``. This must be used with ``query_key``. This will only return a maximum of ``terms_size``,
default 50, unique terms.

``terms_size``: When used with ``use_terms_query``, this is the maximum number of terms returned per query. Default is 50.
Expand Down Expand Up @@ -1370,8 +1303,6 @@ supported by the specified aggregation type. If using a scripted field via ``me

.. note:: When Metric Aggregation has a match, match_body includes an aggregated value that triggered the match so that you can use that on an alert. The value is named based on ``metric_agg_key`` and ``metric_agg_type``. For example, if you set ``metric_agg_key`` to 'system.cpu.total.norm.pct' and ``metric_agg_type`` to 'avg', the name of the value is 'metric_system.cpu.total.norm.pct_avg'. Because of this naming rule, you might face conflicts with jinja2 template, and when that happens, you also can use 'metric_agg_value' from match_body instead.

``doc_type``: Specify the ``_type`` of document to search for.

This rule also requires at least one of the two following options:

``max_threshold``: If the calculated metric value is greater than this number, an alert will be triggered. This threshold is exclusive.
Expand Down Expand Up @@ -1477,9 +1408,7 @@ This rule requires:
``match_bucket_filter``: ES filter DSL. This defines a filter for the match bucket, which should match a subset of the documents returned by the
main query filter.

``doc_type``: Specify the ``_type`` of document to search for.

This rule also requires at least one of the two following options:
ssThis rule also requires at least one of the two following options:

``min_percentage``: If the percentage of matching documents is less than this number, an alert will be triggered.

Expand Down
Loading

0 comments on commit 61777fa

Please sign in to comment.