October 16, 2013 In Engineering By Ritesh Patel

Netflix OSS, meet Docker!

Background

At Nirmata, we are building a cloud services platform to help customers rapidly build cloud ready applications. We believe that next generation of cloud applications will be composed from stateless, loosely-coupled, fine-grained services. In this architecture, each service can be independently developed, deployed, managed and scaled. The Nirmata Platform, itself, is built using the same architectural principles. Such an architecture requires a set of core, infrastructure services. Since Netflix has open sourced their components [1] we decided to evaluate and extend them. The components that best meet our needs were Eureka (a registry for inter-service communication), Ribbon (a client-side load balancer and SDK for service to service communication), Zuul (a gateway service) and Archaius (a configuration framework). In a few days we had the Netflix OSS components working with our services and things looked good!

Challenges in dev/test

Our application now was now made up of six independent services and we could develop and test these services locally on our laptops fairly easily. Next we decided to move our platform to Amazon AWS for testing and integrate with our Jenkins continuous integration server. This required automation to deploy these various services in our test environment. We considered various options such as creating and launching AMI’s, or using Puppet/Chef. But these approaches would require each service to be installed on a separate EC2 instance and the number of EC2 instances would quickly grow as we add more services. Being a startup, we started looking for more efficient alternatives for ourselves as well as our customers..

Why Docker

This is when we started looking at Docker [2]. We knew about Linux containers [3] but didn’t feel we could invest time and effort to directly use them. However, Docker made using Linux containers easy! With a few hours of prototyping with Docker we were able to get an application service up and running. Also, once the Docker images were created, running them was a snap, and unlike launching VM instances there was hardly any startup time penalty. This meant that we could now launch our entire application, very quickly on a single medium EC2 instance vs launching multiple micro or small instances. This was amazing!

Using Netflix OSS with Docker

The next step was to get all our services, including the Netflix OSS services running in Docker (v0.6.3) containers. So, first we created the base image for all our services by installing jdk 7 and tomcat 7. For test environment, we wanted to make sure that the base container image can be use for any service, including Netflix OSS services so we added a short startup script to the container base image to copy the service war file from a mounted location, setup some environment variables (explanation later) and start the tomcat service.

#!/bin/bash

echo Starting $1 on port $2

#Copy the war file from mounted directory to tomcat webapps directory

if [ $1 ]

then

cp /var/lib/webapps/$1.war  /var/lib/tomcat7/webapps/$1.war

fi

#Add the port to the JVM args

if [ $2 ]

then

echo "export JAVA_OPTS=\"-Xms512m -Xmx1024m -Dport.http.nonssl=$2 -Darchaius.configurationSource.additionalUrls=file:///home/nirmata/dynamic.properties\"" >> /usr/share/tomcat7/bin/setenv.sh

else

echo "export JAVA_OPTS=\"-Xms512m -Xmx1024m -Dport.http.nonssl=8080\"" >> /usr/share/tomcat7/bin/setenv.sh

fi

#Setup dynamic properties

echo "eureka.port=$2" >> /home/nirmata/dynamic.properties

if [ $3 ]

then

echo "eureka.serviceUrl.defaultZone=$3" >> /home/nirmata/dynamic.properties

echo "eureka.serviceUrl.default.defaultZone=$3" >> /home/nirmata/dynamic.properties

fi

echo "eureka.environment=" >> /home/nirmata/dynamic.properties

if [ $4 ]

then

echo "default.host=$4" >> /home/nirmata/dynamic.properties

fi

service tomcat7 start

tail -F /var/lib/tomcat7/logs/catalina.out

Following are some key considerations in deployed Netflix OSS with Docker:

Ports – With Docker, ports used by the application in the container need to be specified when launching a container so that the port can be mapped to the host port. Docker automatically assigns the host port. On startup various Nirmata services register with the service registry, Eureka. Service running within a Docker container needs to register using the host port so that other services can communicate with it. To solve this, we specified the same host and container port when launch a container. For example, Eureka would be launched using port 8080 on the container as well as the host. One challenge this introduced was the need to automatically configure Tomcat port for various services. This is easily done by specifying the port as an environment variable and modifying the server.xml file to use the environment variable instead of a hard coded value.

IP Address – Each application typically registers with Eureka using its IP address. We noticed that our services running in Docker containers were registering with Eureka using the loopback (127.0.0.1) IP instead of the container IP. This required a change in the Eureka client code to use the container virtual NIC IP address instead of the loopback IP.

Hostname – Another challenge was hostname resolution. Various Nirmata services register with Eureka using the hostname and IP address but Ribbon just used the hostname to communicate with the services. This proves problematic as there is no DNS service available to resolve the container hostname to IP address. For our existing deployment since Zuul is the only service that communicates with the various other services, we were able to get past this issue by using the same hostname as the Docker host for various service containers (other than the Zuul container). This is not an elegant solution by any means and may not work for all scenarios. My understanding is that Docker 0.7.0 will address this problem with the new links feature.

Dependencies – In our application, there are a few dependencies like each service needs to know the database URL and the Eureka server URL. Docker container IP address is assigned at launch and we couldn’t use the hostname (as described above) to inject this information to our services. We addressed this by launching our services in a predetermined order and by passing in the relevant information at runtime to our container startup script. We used Archaius to load runtime properties dynamically from a file url.

To bring it all together we automated the deployment of our application by developing a basic orchestrator using the docker-java client library. Now we can easily trigger the deployment of entire application from our Jenkins continuous integration server within minutes and test our services on AWS.

Whats next..

The combination of Netflix OSS and Docker makes it really easy to develop, deploy and test distributed applications in the cloud. Our current focus is on building a flexible, application-aware orchestration layer that can make deploying & managing complex applications using Docker that addresses some of the current challenges with using Docker. We would love to hear how you are using Docker, and potentially collaborate on Docker & Netflix OSS related projects.

—

Ritesh Patel

References

[1] Netflix OSS, http://netflix.github.io/

[2] Docker, http://www.docker.io/

[3] LXC, http://en.wikipedia.org/wiki/LXC