ElasticSearch is a popular open source search server that is used for real-time distributed search and analysis of data. When using ElasticSearch for production systems, it should be deployed across multiple servers as a cluster, for the performance, resiliency, and scalability. However, if using public cloud to deploy ElasticSearch, multiple servers means significant additional costs.

In this post, I will demonstrate how Nirmata can be used to deploy ElasticSearch on AWS EC2 spot instances without compromising on performance, resiliency and scalability while reducing costs up to 50%.

First lets look at the cost of deploying a three node ElasticSearch cluster. We will consider a manual deployment as well as using the Amazon ElasticSearch Service. To compare costs, we will use r3.large instances (2 vCPU, 15 GiB Memory)  in all the cases. To keep things simple, we will ignore storage and network bandwidth costs since typically compute costs (EC2 instance) costs tend to be dominant.

Manual Deployment

In case of manual deploy, the user is responsible for installing and managing ElasticSearch. The costs include EC2 instance costs:
One r3.large instance cost = $0.185 per hour or $0.185 x 720 per month
                                                             = $133.2 per month

Cost of three r3.large instances = $399.6 per month

AWS ElasticSearch Service

When using AWS ElasticSearch service, a fully managed service, the costs are inclusive of the EC2 instance costs. For three r3.large.elasticsearch instances:

One r3.large.elasticsearch instance cost = $0.273 per hour or $0.273 x 720 per month
                                                                                          = $196.56 per month

Cost of three r3.large.elasticsearch instances = $589.68 per month

Using Nirmata with EC2 On-demand and Spot Instances

One r3.large on-demand instance cost = $0.185 per hour or $0.185 x 720 per month
                                                                                       = $133.2 per month
Two r3.large on-demand instances cost = 2 x 10% x $0.185 x 720 per month
                                                                                        = $26.64 per month

Cost of three r3.large instances = $159.84 per month

Nirmata usage cost at $2/month/GB for three instances = 4 x 15 x 3 = $180

Total cost using Nirmata & Spot instances =  $133.2 + $26.64 + $180 =  $339.84

Nirmata and EC2 Spot Instances

Now let’s look at how ElasticSearch can be deployed on EC2 spot instances using Nirmata.

1. Nirmata Setup

In Nirmata, we will create two host groups – first one with an on-demand  r3.large EC2 instance. We will also create a Spot Request in EC2 for two r3.large instances. Nirmata integrates with EC2 spot requests, allowing users to create host group for spot instances (see documentation). Create the second host group and use the spot request earlier. Nirmata will discover the spot instances.

Once the host groups are created, you need to create an environment type, let’s call it ElasticSearch and two resource selection policies for the ElasticSearch environment type, with each policy mapping to one of the host groups we created earlier. Make sure both rules have the same rank to ensure that the rules are evaluated together. Also, the rule that maps to the on-demand host group should have minimum placement specified as 1. This will ensure that Nirmata always prefers this host group when placing containers.

2. ElasticSearch Setup

Next, we need to create an application blueprint for ElasticSearch. You can import the ElasticSearch blueprint from the Nirmata catalog and customize it. The environment variables can be modified per your requirements. To ensure resiliency and performance, ElasticSearch should be configured to have 2 shards and 2 replicas. This ensures that the data is replicated across the cluster to prevent any loss of data in case a spot instance abruptly terminates.

3. Deploying ElasticSearch

Once the application blueprint is configured per your requirements, you are ready to deploy the ElasticSearch cluster. Select ElasticSearch environment type and the application blueprint.

Within a few minutes, the entire cluster is deployed.

You can verify that the cluster state is ‘green’ via the ElasticSearch API or by checking the head plugin.

Once the cluster is deployed, it can be actively managed and monitored using Nirmata. You can upgrade ElasticSearch as needed and also use host autoscaling to spin up new on-demand instances in case the spot instances go away.

Cost comparison

	AWS EC2 Instances (Manual)	AWS ElasticSearch Service	Using Nirmata and EC2 Spot Instances
Costs (per month)	$399.6	$589.68	$339.84
Savings using Nirmata	15%	42%	–

 

Summary

As you can see from the cost comparison table above, Using Nirmata with EC2 Sport Instances results in 15% – 40% cost savings compared to other approaches that use only on-demand instances. These cost savings can prove to be significant as your ElasticSearch deployment grows and as you add more clusters. In addition, containerizing ElasticSearch and deploying it using Nirmata provides you with full application lifecycle management capabilities without requiring to build your own tooling. It also gives you portability and flexibility by allowing you to use cheapest or best cloud resources depending on your use case e.g. dev/test vs production.

Try Nirmata now with a 30-day free trial!

Download eBook: Containerizing Traditional Applications