We want custom high-speed background workers!

Log archives and metrics would be nice.

Ruby's too slow!

Orchestration

Instance Management

Deployment

Monitoring & Logging

Orchestration

• Which instances are running which apps?
• Need to create logical, addressable groups of instances.
• Netflix's Asgard

Instance Management

• Setup EC2 instances to run apps.
• Chef, Puppet, etc.

Deployment

• Package-based? Source-based?
• Capistrano

Monitoring & Logging

• Real-time metrics for system and application health.
• Archives for troubleshooting.
• Nagios, Sensu
• Graphite, Librato
• l2met

OpsWorks gives us logical containers: stacks and layers.

These map nicely to Heroku apps and Procfiles.

$ cat Procfile

web:      bundle exec unicorn -c ./config/unicorn.rb
worker:   bundle exec rake resque:work INTERVAL=0.1 QUEUE=*
consumer: bundle exec rake consumer

$ aws opsworks describe-stacks
{
  "Stacks": [
    {
      "Name": "groupme",
      "StackId": "XXXXXX",
      ...

$ aws opsworks describe-instances --stack-id XXXXXX
{
  "Instances": [
    {
      "PublicDns": "ec2-55-55-555-555.compute-1.amazonaws.com",
      ...

Since OpsWorks holds the running state and is easily queried, there's no need to synchronize this data anywhere.

This was a great design decision espoused by Netflix's Asgard.

OpsWorks is pre-baked with Chef.

We provide custom cookbooks.

• rvm & ruby
• node.js and npm
• nginx
• syslog-ng

Chef is only run at setup and teardown.

Immutable Infrastructure

1. Upload formation

The mapping is maintained by a little app called meta.

Basically, a simple REST interface on top of DynamoDB.

1. Upload formation

$ curl https://meta/apps/show/groupme
{
  "formation.app":      "web=1",
  "formation.worker":   "worker=4",
  "formation.consumer": "consumer=1"
  ...
}

$ echo "web=1" > /etc/app/formation

2. Upload config vars

Once again, meta is in charge of this.

$ curl https://meta/config_vars/show/groupme
{
  "RAILS_ENV":    "production",
  "REDIS_URL":    "redis://ec2-44-44-444-444.amazonaws.com:6479/0",
  "DATABASE_URL": "postgres://user:secret@ec2-66-66-666-666.amazonaws.com:5432/",
  ...
}

$ echo "RAILS_ENV=..." > /etc/app/env

3. Update code

3. Update code

First, a pretty traditional remote git repo.

$ git fetch origin && git reset --hard $REF

3. Update code

And also, a Heroku-style slug-based deploy.

$ curl -s -o /tmp/SLUG https://anvil/slugs/SLUG
$ tar x -C /tmp/app /tmp/SLUG

4. Start it up

exec su groupme -l -s /bin/bash -c  \
  "rvm-exec default foreman start   \
     -p $PORT                       \
     -f /tmp/app/Procfile           \
     -e /etc/app/env                \
     -m $(cat /etc/app/formation) 2>&1" >> /var/log/app/app.log

$ gun deploy
$ gun console
$ gun console.db
$ gun status
$ gun instances
$ gun logs
$ gun ps.restart
$ gun ssh.host
$ gun git.compare

Inspired by l2met, this is an EventMachine HTTP server that parses syslog-formatted logs and extracts metrics.

Those metrics are fed to statsd, and ultimately, graphite.

CopperEgg

PagerDuty

New Relic

A few seconds

$ heroku scale web=10 -a groupme

A few minutes

$ gun app:groupme instances.create:app, \
    count=3,                            \
    instance_type=c1.xlarge,            \
    availability_zone=us-east-1a

• Docker
• EC2 Backplane
• Software Delivery via Amazon Docker Workflow by Ryan Smith

• ElastiCache
• VPC
• IAM Roles
• DirectConnect

• CI Integration
• Chat Ops