Distribute Watches in Elasticsearch

X-Pack Alerting aka Elasticsearch Watcher can run distributed in a Elasticsearch cluster.

The basic concept: The watcher index .watches or .watches-6 contains all watches. If you increase the replicas, multiple elasticsearch nodes, that contains the index shard (replica), are used by the master node for alerting.

On each node, that contains one of the .watches shards, a scheduler, .. runs. .. The more replica shards you add, the more distributed the watches can be executed.

(Source: https://www.elastic.co/guide/en/x-pack/6.2/how-watcher-works.html)

To increase the replica size:

PUT .watches-6/_settings
  "number_of_replicas": 2

With the primary shard and 2 replicas, you should have three elasticsearch nodes that run your watchers.

You can check it by querying all thread pools:

GET _cat/thread_pool?v&s=name:desc&h=host,name,active,rejected,completed

The watcher thread pool will appear on top of the results.

host             name                active rejected completed
alpha            watcher                  0        0         0
master1          watcher                  0        0         0
gamma            watcher                  0        0     12668
delta            watcher                  0        0      4275
beta             watcher                  0        0         0
master2          watcher                  0        0         0
master3          watcher                  0        0         0
omega            watcher                  0        0     15693