FAQ

Shortcuts for subsequent ES commands

export ES_URL=https://haraka:pass@elasticsearch:9200
export ESA_URL=https://admin:pass@elasticsearch:9200

---

E: maximum normal shards open

this cluster currently has [1000]/[1000] maximum normal shards open

When you see the above error message in your logs, you need to:

- reduce the number of shards in your cluster
- increase your clusters shard limits.

---

Q: How to increase the cluster shards

A1: Add another host to the cluster

A2: Raise the shard limit

curl -XPUT "$ESA_URL/_cluster/settings" -H 'Content-Type: application/json' -d'
{ "persistent" : { "action.search.shard_count.limit" : "4500" } }'

---

Q: How to reduce the number of shards

A: Roll up old indexes

The following commands rolls up the April 2022 daily indexes into a single monthly index and then deletes the daily indexes. That reduces the number of shards by a factor of 30.

curl -XPOST --insecure "$ES_URL/_reindex?pretty" -H 'Content-Type: application/json' -d'
{
    "conflicts": "proceed",
    "source": {
        "index": "smtp-transaction-2022-04-*"
    },
    "dest": {
        "index": "smtp-transaction-2022-04"
    }
}'

curl -XDELETE --insecure "$ESA_URL/smtp-transaction-2022-04-*"

Newer versions of ES require this setting to be changed so that admins can delete daily indexes with a wildcard:

curl -XPUT --insecure "$ESA_URL/_cluster/settings" -H 'Content-type: application/json' -d'
{ "persistent" : { "action.destructive_requires_name": false }}'

---

Q: How to increase total fields

curl -XPUT "$ESA_URL/smtp-*/_settings" -H 'Content-Type: application/json' -d'
{
    "index" : {
        "mapping" : {
            "total_fields" : {
                "limit" : "100000"
            }
        }
    }
}'