raczylo.com blog ~ #

Elasticsearch, AWS in different availability zones

Created on 2014-09-15 in categories ops , cloud , tutorial
Tagged as: aws , elasticsearch , cloud , cluster

If you have troubles with clustering your Elasticsearch instances between different AWS availability zones

Quick tutorial to save your time and nerves. If you’re in real hurry - copy / paste and follow further instructions.

Before proceeding - don’t forget to allow communication between zones on ports 9200 and 9300 in EC2 security groups ( unless all hosts are using the same VPC ).

apt-get update
apt-get install elasticsearch
/usr/share/elasticsearch/bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.10.0
# this should take max 3 seconds, if longer - ctrl+c and retry
mkdir -p /ebs/elasticsearch
chown elasticsearch:elasticsearch /ebs/elasticsearch

Next step - paste into default config ( /etc/elasticsearch/elasticsearch.yml ):

cluster.name: mycluster
cloud.aws.access_key: ****
cloud.aws.secret_key: ****
cloud.aws.region: eu-west
discovery.type: ec2
discovery.ec2.host_type: private_ip
discovery.ec2.ping_timeout: 5s
discovery.ec2.tag.Environment: Staging
node.name: my-fancy-node-name-staging
path.conf: /etc/elasticsearch
path.data: /ebs/elasticsearch

That’s where the real fun begin. To understand how to cluster elasticsearch across the regions let’s assume that you have four different instances ( 2 per zone ) in zones eu-west and us-east. Keep in mind that you should add ‘environment’:‘staging’ tag to make it work ( or this example at least ).

eu-inst-1      us-inst-1
eu-inst-2      us-inst-1

In this case communication scheme should look like this:

eu-inst-1 «---------» us-inst-1
   ↑                     ↑
   ↓                     ↓
eu-inst-2             us-inst-2
  • eu-inst-1 should search for other hosts in us-east region
  • us-inst-1 should search for other hosts in eu-east region
  • eu-inst-2 and us-inst-2 should search in their own regions

Remember to change region configuration on each of those hosts. Proper values are:

us-east (us-east-1), us-west (us-west-1), us-west-1, us-west-2, ap-southeast (ap-southeast-1), ap-southeast-1, ap-southeast-2, ap-northeast (ap-northeast-1) eu-west (eu-west-1).

After restarting elasticsearch on all hosts they should elect one master and rest of them ( including those in different availability zone ) should automatically join the cluster. To check current cluster status login to one of the hosts and execute following command:

curl localhost:9200/_cluster/state


* Table of contents *

* Check other posts *

* Categories *

aws(1) cloud(2) dev(2) gems(1) howto(3) mac(1) ops(7) projects(2) script(1) tools(1) tutorial(4)

* Tags *

api(1) apple(2) aws(3) bash(1) cloud(4) cluster(1) ec2(1) elasticsearch(1) gem(1) github(1) google-authenticator(1) google-cloud(1) iptables(1) linux(2) logging(1) mac(2) opensource(1) openvpn(1) osx(2) pci-compliance(1) ruby(2) security(2) ssh(1) ubuntu(1) wykop(1) yubikey(1)
comments powered by Disqus