As enterprises adopt containers and microservices, their motivations for adopting these technologies vary depending on their current operational challenges as well as their future goals. For example, some enterprises may be seeking agility and increase frequency of deployments by adopting microservices architecture, while other organizations may be interested in saving costs by increasing utilization of their servers by leveraging containers.

In fact, in larger enterprises, different teams typically adopt newer technologies at a different pace and their needs and motivations vary. As a result it is not uncommon to see teams within an enterprise using different clouds or different schedulers for deploying their applications. At Nirmata we have always supported multiple clouds, both public and private. Some of our users have expressed the requirement to support external schedulers with Nirmata. In this post, I will discuss how to use Nirmata to deploy and manage an Apache Mesos environment on any cloud.

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks. Mesos is an open source project originally developed at the University of California at Berkeley. It sits between the application layer and the operating system and makes it easier to deploy and manage applications in large-scale clustered environments more efficiently. It can run many applications on a shared pool of nodes. A Mesos deployment consists of master and slave daemons, and frameworks. A framework running on top of Mesos consists of two components: a scheduler that registers with the master to be offered resources, and an executor process that is launched on agent nodes to run the framework’s tasks. Some popular Mesos frameworks are Marathon, Chronos and Aurora. You can see a list of Mesos frameworks here.

Marathon is an open source, Apache Mesos framework for container orchestration. Marathon provides a REST API for starting, stopping, and scaling applications. Marathon can also be used to start other Mesos frameworks, and it can also launch any process that can be started in the regular shell.

Source: http://datastrophic.io/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka/

Deploying Mesos + Marathon

To deploy a Mesos cluster using Nirmata, you need to first create a blueprint. You can import the existing Nirmata catalog item for Mesos + Marathon. The blueprint contains four services – Zookeeper, Mesos master, Mesos slave and Marathon and includes predefined settings for configuring and deploying these services but you can adjust these settings as need. For example, the blueprint uses service affinity rules to ensure that Zookeeper, Mesos master and Marathon are deployed in the same host while Mesos slave is always deployed on a unique host (i.e. on a different host all by itself) but these can be changed per your setup.

Prior to deploying the blueprint, you need to onboard your cloud resources and setup resource selection policies. Nirmata’s Resource selection policies provide tremendous flexibility in configuring your physical and virtual resources. For example you can create a resource selection rule to deploy your Mesos slaves on different types of hosts, perhaps across different regions in a cloud provider, or even use a combination of on-demand instances and spot instances on AWS for your Mesos slaves while using on-demand instances for zookeeper, Mesos master and Marathon.

Managing Mesos Cluster

Once your Mesos environment is deployed, your cluster is up and running and available to schedule tasks. With Nirmata, you can continue to monitor your cluster and get visibility into which tasks are running on each host. As the utilization of the Mesos cluster increases, Nirmata will not only generate alarms to indicate high CPU/memory consumption but you can also setup host auto scaling policy to add new slaves to your cluster.

In addition, Nirmata also simplifies the process of upgrading the Mesos cluster. You can perform a controlled upgrade of Zookeeper and Mesos Master followed by a rolling upgrade of the Mesos slaves to minimize any down time.  

Summary

Nirmata simplifies deployment and management of complex applications. You can now deploy and manage Mesos clusters with just a few clicks using Nirmata. This provides tremendous flexibility but allowing you to set up isolated clusters not only for different teams but also for different phases of your DevOps pipeline. You also have the ability to select schedulers appropriate for your application. For example you can use Mesos to schedule short-lived tasks or big data jobs while using native Nirmata orchestration for long lived services as well as stateful services.

As we continue to add support for popular schedulers and cluster managers in Nirmata, we would love to hear more about the challenges you face while deploying and managing your applications. Please try out Nirmata and let us know your feedback.

For more updates and news follow us at: