The Show Must Go On

When preparing for a big event, it is our job to make sure the general public sees exactly what is expected, and with the help of Amazon Web Service (AWS) we did! All planning comes with a few standard issue assessments/steps: Identify need, identify options, and begin to build!

Identifying Need:
Metal Toad Media has hosted the Emmy’s website for four years, and every year in preparation for the Emmy’s big events; during nominations (July 18, 2013) and the live event (September 22, 2013), Metal Toad Media prepared by building out more servers to handle the large traffic increase. This past year, between nominations and the live event, Dsire was given the monumental task of rebuilding the Emmy’s website - read about their adventure here.

In September, we were notified Dsire had finished the new site and new code base. The site would experience the same traffic hikes during the live event as seen during nominations, thus, we ran our load test to ensure the new site would perform as expected. Clearance to do the test came a week before the event, and the new requirements were much more CPU intensive than originally anticipated. During the test we found the CPU load was 8:1 (server process requirements were 8x higher than what we had provisioned).

To give an idea of the traffic spike, before nominations the standard was a few thousand hits a day. During the live event, there were over 3.5 million page views and 220 million http hits in a 3-hour period.

Identifying Options
Metal Toad insures 100% up-time of the website for the Emmy live event, so with test results expressing urgent need, and four days time, we had three options:

• Create a landing page that would not crash (a non-professional option that would contractually work)
• Get more servers (which would allow us to be prepared, but would cost tens of thousands in equipment that would not be utilized anytime after the event, and the set-up time would take far more than the four days we had until the event)
• Or, host using AWS and build a custom cloud for the client’s needs. In this instance, AWS cloud offered several advantages:
• More flexibility to scale up and down during events
• Redundancy in multiple physical locations (availability zones)
• Avoids costs of a second move back to co-location

Building a Custom Cloud



When we determined constructing an AWS custom cloud for Emmys.com would be the best solution, we started where we lacked the most, processing power for the web app. By using the performance data from our before and after tests to calculate the number of CPU’s we would need, we found that 11 Web servers each with 8 CPUs and 30GB of RAM were necessary; these would be the backbone of the Drupal site, running PHP, apache, glusterfs, and Memcache.

Since Drupal is a database heavy application, we also needed to move the database server to AWS. Instead of spinning up more EC2 instances and managing this manually, we leveraged their RDS. This provisioned the database service and a slave server in a separate availability zone, each was configured with 8 CPU, 65GB of ram, and had 20,000 reserved IOPS.

Next, we needed a way to balance traffic among the webheads. To do this we used AWS Elastic Load Balancer which allows us to balance traffic between web servers and in the event that one failed, it would be removed from circulation.

Lastly, we weren’t going to be able to build, configure, and maintain these servers by hand. So, we leveraged our existing Puppet master and modified the classes to work with AWS, this allowed us to spin up the AWS Instances in minutes instead of hours or days.

The End Result
As Vice President Tony put it: “After studying the stats from the Creative Arts show, we realized substantially more hardware was needed. In less than four days, our devops team built a new cloud stack, and migrated all three sites plus a new mobile domain, and the iPhone data feeds.”

It was a great feeling to see the Emmy’s website survive the rush of the 65th annual live show. Metal Toad found a strong utilization for custom cloud building, and the general public got a reminder that if Neil Patrick Harris ever does Dancing with the Stars, no one stands a chance.