Add Elasticsearch 8 support + remove ES 6 support

CHANGELOG.md

@@ -12,12 +12,16 @@
 # 2.x.x
 
 ## Breaking changes
+- Add support for Elasticsearch 8, remove support for Elasticsearch 6 and below - [#744](https://github.com/jertel/elastalert2/pull/744) - @ferozsalam, @jertel, and @nsano-rururu
+  WARNING! Read the [ES 8 upgrade notes](https://elastalert2.readthedocs.io/en/latest/recipes/faq.html#does-elastalert-2-support-elasticsearch-8) BEFORE upgrading your cluster to Elasticsearch 8. Failure to do so can result in your cluster no longer starting and unable to rollback to 7.x.
+- Kibana dashboard integration has been removed, as it only was supported with older versions of Elasticsearch and Kibana. Per the above breaking change those older versions are no longer supported by ElastAlert 2.
 - Dockerfile refactor for app home and user home to be the same directory (/opt/elastalert/). Before app home is /opt/elastalert/ and user home is /opt/elastalert/elastalert. After app home and user home are the same /opt/elastalert/ - [#656](https://github.com/jertel/elastalert2/pull/656)
 
 ## New features
 - [MS Teams] Kibana Discover URL and Facts - [#660](https://github.com/jertel/elastalert2/pull/660) - @thib12
 - Add support for Kibana 7.17 for Kibana Discover - [#695](https://github.com/jertel/elastalert2/pull/695) - @nsano-rururu
 - Added a fixed name metric_agg_value to MetricAggregationRule match_body - [#697](https://github.com/jertel/elastalert2/pull/697) - @iamxeph
+
 ## Other changes
 - Load Jinja template when loading an alert - [#654](https://github.com/jertel/elastalert2/pull/654) - @thib12
 - tox 3.24.4 to 3.24.5 - [#655](https://github.com/jertel/elastalert2/pull/655) - @nsano-rururu

chart/elastalert2/values.yaml

@@ -79,7 +79,6 @@ extraConfigOptions: {}
   # # Options to propagate to all rules, e.g. a common slack_webhook_url or kibana_url
   # # Please note at the time of implementing this value, it will not work for required_locals
   # # Which MUST be set at the rule level, these are: ['alert', 'type', 'name', 'index']
-  # generate_kibana_link: true
   # kibana_url: https://kibana.yourdomain.com
   # slack_webhook_url: dummy
 

docs/source/elastalert.rst

@@ -66,7 +66,7 @@ Additional rule types and alerts can be easily imported or written. (See :ref:`W
 
 In addition to this basic usage, there are many other features that make alerts more useful:
 
-- Alerts link to Kibana dashboards
+- Alerts link to Kibana Discover searches
 - Aggregate counts for arbitrary fields
 - Combine alerts into periodic reports
 - Separate alerts by using a unique key field

docs/source/recipes/faq-md.md

@@ -219,8 +219,7 @@ rounded to a single timestamp.
 
 If you are using ``query_key`` (a single key, not multiple keys) you can use ``use_terms_query``.
 This will make ElastAlert 2 perform a terms aggregation to get the counts for each value of a certain
-field. Both ``use_terms_query`` and ``use_count_query`` also require ``doc_type`` to be set to the
-``_type`` of the documents. They may not be compatible with all rule types.
+field. May not be compatible with all rule types.
 
 Can I perform aggregations?
 ==========
@@ -436,21 +435,20 @@ filter:
 Does ElastAlert 2 support Elasticsearch 8?
 ===========
 
-Support for Elasticsearch 8 is a work in progress. It is currently possible to
-load ElastAlert 2 against a _fresh_ installation of Elasticsearch (i.e. one where
-no previous ElastAlert instance has been running) without any extra steps.
+ElastAlert 2 supports Elasticsearch 8.
 
 To upgrade an existing ElastAlert 2 installation to Elasticsearch 8 the
-following manual steps are required:
+following manual steps are required (note the important WARNING below):
 
 * Shutdown ElastAlert 2.
-* Delete or rename the old `elastalert*` indices. See Elasticsearch
-  documentation for instructions on how to delete via the API.
+* Delete the old `elastalert*` indices. See [Elasticsearch
+  documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html)
+  for instructions on how to delete via the API, or use the Kibana Index Management interface.
+* Upgrade the Elastic cluster to Elasticsearch 8.
 * If NOT running ElastAlert 2 via Docker or Kubernetes, run
   elastalert-create-index to create the new indices. This is not needed when
   running via a container since the container always attempts to creates the
   indices at startup, if they're not yet created.
 * Restart ElastAlert 2.
 
-However, *at this point it is not guaranteed that features which used to work
-on Elasticsearch 7 will still work*.
+WARNING: Failure to remove the old ElastAlert indices can result in a non-working Elasticsearch cluster. This is because the ElastAlert indices contain deprecated features and the Elasticsearch 8 upgrade logic is currently flawed and does not correctly handle this situation. The Elasticsearch GitHub repository contains [more information](https://github.com/elastic/elasticsearch/issues/84199) on this problem.

docs/source/recipes/writing_filters.rst

@@ -100,7 +100,7 @@ For ranges on fields::
 Negation, and, or
 *****************
 
-For Elasticsearch 2.X, any of the filters can be embedded in ``not``, ``and``, and ``or``::
+Any of the filters can be embedded in ``not``, ``and``, and ``or``::
 
     filter:
     - or:

docs/source/ruletypes.rst

@@ -60,22 +60,12 @@ Rule Configuration Cheat Sheet
 +--------------------------------------------------------------+           |
 | ``description`` (string, default empty string)               |           |
 +--------------------------------------------------------------+           |
-| ``generate_kibana_link`` (boolean, default False)            |           |
-+--------------------------------------------------------------+           |
-| ``use_kibana_dashboard`` (string, no default)                |           |
-+--------------------------------------------------------------+           |
 | ``kibana_url`` (string, default from es_host)                |           |
 +--------------------------------------------------------------+           |
 | ``kibana_username`` (string, no default)                     |           |
 +--------------------------------------------------------------+           |
 | ``kibana_password`` (string, no default)                     |           |
 +--------------------------------------------------------------+           |
-| ``use_kibana4_dashboard`` (string, no default)               |           |
-+--------------------------------------------------------------+           |
-| ``kibana4_start_timedelta`` (time, default: 10 min)          |           |
-+--------------------------------------------------------------+           |
-| ``kibana4_end_timedelta`` (time, default: 10 min)            |           |
-+--------------------------------------------------------------+           |
 | ``generate_kibana_discover_url`` (boolean, default False)    |           |
 +--------------------------------------------------------------+           |
 | ``shorten_kibana_discover_url`` (boolean, default False)     |           |
@@ -177,13 +167,9 @@ Rule Configuration Cheat Sheet
 | ``attach_related`` (boolean, default False)           |        |           |           |        |    Opt    |       |          |        |           |                  |                 |                |
 +-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
 |``use_count_query`` (boolean, default False)           |        |           |           |        |     Opt   | Opt   | Opt      |        |           |                  |                 |                |
-|                                                       |        |           |           |        |           |       |          |        |           |                  |                 |                |
-|``doc_type`` (string, no default)                      |        |           |           |        |           |       |          |        |           |                  |                 |                |
 +-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
 |``use_terms_query`` (boolean, default False)           |        |           |           |        |     Opt   | Opt   |          | Opt    |           |                  |                 |                |
 |                                                       |        |           |           |        |           |       |          |        |           |                  |                 |                |
-|``doc_type`` (string, no default)                      |        |           |           |        |           |       |          |        |           |                  |                 |                |
-|                                                       |        |           |           |        |           |       |          |        |           |                  |                 |                |
 |``query_key`` (string or list, no default)             |        |           |           |        |           |       |          |        |           |                  |                 |                |
 |                                                       |        |           |           |        |           |       |          |        |           |                  |                 |                |
 |``terms_size`` (int, default 50)                       |        |           |           |        |           |       |          |        |           |                  |                 |                |
@@ -224,8 +210,6 @@ Rule Configuration Cheat Sheet
 |                                                       |        |           |           |        |           |       |          |        |           |                  |                 |                |
 |([min|max|avg|sum|cardinality|value_count|percentiles])|        |           |           |        |           |       |          |        |           |                  |                 |                |
 +-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
-|``doc_type`` (string, no default)                      |        |           |           |        |           |       |          |        |           |  Req             |  Req            |  Req           |
-+-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
 |``metric_agg_script`` (no default)                     |        |           |           |        |           |       |          |        |           |  Opt             |  Opt            |                |
 +-------------------------------------------------------+--------+-----------+-----------+--------+-----------+-------+----------+--------+-----------+------------------+-----------------+----------------+
 |``percentile_range`` ++required if percentiles is used |        |           |           |        |           |       |          |        |           |  Req++           |  Req++          |                |
@@ -596,27 +580,13 @@ description
 ``description``: text describing the purpose of rule. (Optional, string, default empty string)
 Can be referenced in custom alerters to provide context as to why a rule might trigger.
 
-generate_kibana_link
-^^^^^^^^^^^^^^^^^^^^
-
-``generate_kibana_link``: This option is for Kibana 3 only.
-If true, ElastAlert 2 will generate a temporary Kibana dashboard and include a link to it in alerts. The dashboard
-consists of an events over time graph and a table with ``include`` fields selected in the table. If the rule uses ``query_key``, the
-dashboard will also contain a filter for the ``query_key`` of the alert. The dashboard schema will
-be uploaded to the kibana-int index as a temporary dashboard. (Optional, boolean, default False)
-
 kibana_url
 ^^^^^^^^^^
 
 ``kibana_url``: The base url of the Kibana application. If not specified, a URL will be constructed using ``es_host``
 and ``es_port``.
 
-This value will be used if one of the following conditions are met:
-
-- ``generate_kibana_link`` is true
-- ``use_kibana_dashboard`` is true
-- ``use_kibana4_dashboard`` is true
-- ``generate_kibana_discover_url`` is true and ``kibana_discover_app_url`` is a relative path
+This value will be used if ``generate_kibana_discover_url`` is true and ``kibana_discover_app_url`` is a relative path
 
 (Optional, string, default ``http://<es_host>:<es_port>/_plugin/kibana/``)
 
@@ -636,38 +606,6 @@ This value is only used if ``shorten_kibana_discover_url`` is true.
 
 (Optional, string, no default)
 
-use_kibana_dashboard
-^^^^^^^^^^^^^^^^^^^^
-
-``use_kibana_dashboard``: The name of a Kibana 3 dashboard to link to. Instead of generating a dashboard from a template,
-ElastAlert 2 can use an existing dashboard. It will set the time range on the dashboard to around the match time,
-upload it as a temporary dashboard, add a filter to the ``query_key`` of the alert if applicable,
-and put the url to the dashboard in the alert. (Optional, string, no default)
-
-use_kibana4_dashboard
-^^^^^^^^^^^^^^^^^^^^^
-
-``use_kibana4_dashboard``: A link to a Kibana 4 dashboard. For example, "https://kibana.example.com/#/dashboard/My-Dashboard".
-This will set the time setting on the dashboard from the match time minus the timeframe, to 10 minutes after the match time.
-Note that this does not support filtering by ``query_key`` like Kibana 3.  This value can use `$VAR` and `${VAR}` references
-to expand environment variables.
-
-kibana4_start_timedelta
-^^^^^^^^^^^^^^^^^^^^^^^
-
-``kibana4_start_timedelta``: Defaults to 10 minutes. This option allows you to specify the start time for the generated kibana4 dashboard.
-This value is added in front of the event. For example,
-
-``kibana4_start_timedelta: minutes: 2``
-
-kibana4_end_timedelta
-^^^^^^^^^^^^^^^^^^^^^
-
-``kibana4_end_timedelta``: Defaults to 10 minutes. This option allows you to specify the end time for the generated kibana4 dashboard.
-This value is added in back of the event. For example,
-
-``kibana4_end_timedelta: minutes: 2``
-
 generate_kibana_discover_url
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -735,9 +673,8 @@ kibana_discover_version
 
 The currently supported versions of Kibana Discover are:
 
-- `5.6`
-- `6.0`, `6.1`, `6.2`, `6.3`, `6.4`, `6.5`, `6.6`, `6.7`, `6.8`
 - `7.0`, `7.1`, `7.2`, `7.3`, `7.4`, `7.5`, `7.6`, `7.7`, `7.8`, `7.9`, `7.10`, `7.11`, `7.12`, `7.13`, `7.14`, `7.15`, `7.16`, `7.17`
+- `8.0`
 
 ``kibana_discover_version: '7.15'``
 
@@ -747,6 +684,8 @@ kibana_discover_index_pattern_id
 ``kibana_discover_index_pattern_id``: The id of the index pattern to link to in the Kibana Discover application.
 These ids are usually generated and can be found in url of the index pattern management page, or by exporting its saved object.
 
+In this documentation all references of "index pattern" refer to the similarly named concept in Kibana 8 called "data view".
+
 Example export of an index pattern's saved object:
 
 .. code-block:: text
@@ -1106,12 +1045,10 @@ Optional:
 
 ``use_count_query``: If true, ElastAlert 2 will poll Elasticsearch using the count api, and not download all of the matching documents. This is
 useful is you care only about numbers and not the actual data. It should also be used if you expect a large number of query hits, in the order
-of tens of thousands or more. ``doc_type`` must be set to use this.
-
-``doc_type``: Specify the ``_type`` of document to search for. This must be present if ``use_count_query`` or ``use_terms_query`` is set.
+of tens of thousands or more.
 
 ``use_terms_query``: If true, ElastAlert 2 will make an aggregation query against Elasticsearch to get counts of documents matching
-each unique value of ``query_key``. This must be used with ``query_key`` and ``doc_type``. This will only return a maximum of ``terms_size``,
+each unique value of ``query_key``. This must be used with ``query_key``. This will only return a maximum of ``terms_size``,
 default 50, unique terms.
 
 ``terms_size``: When used with ``use_terms_query``, this is the maximum number of terms returned per query. Default is 50.
@@ -1241,12 +1178,10 @@ cause alerts. Baseline is established after ``timeframe`` has elapsed twice sinc
 
 ``use_count_query``: If true, ElastAlert 2 will poll Elasticsearch using the count api, and not download all of the matching documents. This is
 useful is you care only about numbers and not the actual data. It should also be used if you expect a large number of query hits, in the order
-of tens of thousands or more. ``doc_type`` must be set to use this.
-
-``doc_type``: Specify the ``_type`` of document to search for. This must be present if ``use_count_query`` or ``use_terms_query`` is set.
+of tens of thousands or more. 
 
 ``use_terms_query``: If true, ElastAlert 2 will make an aggregation query against Elasticsearch to get counts of documents matching
-each unique value of ``query_key``. This must be used with ``query_key`` and ``doc_type``. This will only return a maximum of ``terms_size``,
+each unique value of ``query_key``. This must be used with ``query_key``. This will only return a maximum of ``terms_size``,
 default 50, unique terms.
 
 ``terms_size``: When used with ``use_terms_query``, this is the maximum number of terms returned per query. Default is 50.
@@ -1268,12 +1203,10 @@ Optional:
 
 ``use_count_query``: If true, ElastAlert 2 will poll Elasticsearch using the count api, and not download all of the matching documents. This is
 useful is you care only about numbers and not the actual data. It should also be used if you expect a large number of query hits, in the order
-of tens of thousands or more. ``doc_type`` must be set to use this.
-
-``doc_type``: Specify the ``_type`` of document to search for. This must be present if ``use_count_query`` or ``use_terms_query`` is set.
+of tens of thousands or more.
 
 ``use_terms_query``: If true, ElastAlert 2 will make an aggregation query against Elasticsearch to get counts of documents matching
-each unique value of ``query_key``. This must be used with ``query_key`` and ``doc_type``. This will only return a maximum of ``terms_size``,
+each unique value of ``query_key``. This must be used with ``query_key``. This will only return a maximum of ``terms_size``,
 default 50, unique terms.
 
 ``terms_size``: When used with ``use_terms_query``, this is the maximum number of terms returned per query. Default is 50.
@@ -1370,8 +1303,6 @@ supported by the specified aggregation type.  If using a scripted field via ``me
 
 .. note:: When Metric Aggregation has a match, match_body includes an aggregated value that triggered the match so that you can use that on an alert. The value is named based on ``metric_agg_key`` and ``metric_agg_type``. For example, if you set ``metric_agg_key`` to 'system.cpu.total.norm.pct' and ``metric_agg_type`` to 'avg', the name of the value is 'metric_system.cpu.total.norm.pct_avg'. Because of this naming rule, you might face conflicts with jinja2 template, and when that happens, you also can use 'metric_agg_value' from match_body instead.
 
-``doc_type``: Specify the ``_type`` of document to search for.
-
 This rule also requires at least one of the two following options:
 
 ``max_threshold``: If the calculated metric value is greater than this number, an alert will be triggered. This threshold is exclusive.
@@ -1477,9 +1408,7 @@ This rule requires:
 ``match_bucket_filter``: ES filter DSL. This defines a filter for the match bucket, which should match a subset of the documents returned by the
 main query filter.
 
-``doc_type``: Specify the ``_type`` of document to search for.
-
-This rule also requires at least one of the two following options:
+ssThis rule also requires at least one of the two following options:
 
 ``min_percentage``: If the percentage of matching documents is less than this number, an alert will be triggered.