No more pets! Releasing cattle to the Oracle Public Cloud

4 November 2018 · opc

At the dawn of the 2010s, a number of cloud thought leaders were looking for a good way to explain to customers how cloud providers, cloud native apps, and cloud more generally were fundamentally different from what had come and gone before. For cloud pioneer Randy Bias, there was a light that went on above his head when he realised he had found the perfect elevator pitch.

Cloud infrastructure is like cattle, we no longer think of our servers as pets.

If that’s a bit cryptic for you let us lift a direct quote.

“In the old way of doing things, we treat our servers like pets, for example Bob the mail server. If Bob goes down, it’s all hands on deck. The CEO can’t get his email and it’s the end of the world. In the new way, servers are numbered, like cattle in a herd. For example, www001 to www100. When one server goes down, it’s taken out back, shot, and replaced on the line.” - Randy Bias, circa 2011

This metaphor would become the go to story for framing all kinds of concepts and technologies from Immutable Infrastructure to Cloud Native Apps to Containers to Kubernetes and everything between.

In this article, I’m going to put the metaphor into real terms by clearly laying out the raw ingredients for Immutable Infrastructure on Oracle Public Cloud which I’ve intentionally picked for our example, as it’s a lesser known but albeit powerful player in the ever growing list of Infrastructure-as-a-Service upstarts rivaling the AWS trailblazing of the 2010s.

If the Pets vs Cattle story is so fundamentally useful for defining Cloud concepts such as Immutable Infrastructure, then it should also be true that the ingredients of said Immutable Infrastructure should be applicable to any Public Cloud provider that is worth the investment.

The pain and glory of SSHing

When our servers were pets we would be constantly maintaining them by SSHing on to the box and pampering them until they became old and perhaps a little worn down and confused; a bit like the much loved family Shitzu in their penultimate years.

SSHing, the act of connecting to a Linux server over the SSH protocol in order to view or change its state, is not bad per se. But it’s not great either. Every person that SSHes onto a long-lived server leaves an opinion behind: a hand-edited config file here, a hot-fixed library there, a cheeky cowsay in the login banner that nobody remembers adding. Multiply that by years of different owners and you end up with a server whose true state lives nowhere except on the server itself. Nobody can rebuild it, so everybody is afraid of it. That is how a pet becomes a liability.

Wouldn’t it be great if we could start over?

When your server is maintained by SSHing and it all goes bad…

  1. Do you frantically look back over some notes to see what was previously changed on the server?
  2. Do you SSH back in and try to diagnose and fix the problem in place?
  3. Or, do you destroy the whole thing and start over?

If the third point terrifies you, this article is for you.

So let’s say for the sake of argument our server is basically burnt to the ground. Now, are we going to be happy about building it back from the ground up? Probably not right?

But what if we could bring it back at the push of a button, including every little minor detail that we had applied to it over the years?

Of course it would soften the blow… considerably so.

All of this is possible thanks to a concept known as baking.

Nothing like a mixed metaphor

Baking is nothing new, Netflix have been doing this on AWS for over a decade. Despite that though, to this day while it is very popular in many circles, it is not yet a universally adopted approach… that won’t be for long, immutable infrastructure is a much greater paradigm than SSHing or changing servers on the fly.

Have you ever seen how a large-scale bakery handles the production of its daily goods? Of course they don’t put their sponge cake in the oven only to take it back out, manipulate it and then put it back in again. They bake it to a strict recipe and then it’s consumed by many. Now unlike baking a cake, with a machine image you really can bake it once and eat it as many times as you’d like! And by eat, I of course mean deploy to a Cloud Infrastructure at scale so that it can serve up some value to the end user… rolls off the tongue.

The recipe is source controlled, the oven is an image builder such as HashiCorp Packer, and the baked good is a machine image that can be stamped out into as many identical, disposable instances as we need. Servers stop being pets we nurse back to health and become cattle we replace from the herd.

Entering the cattle farm

So without further ado, let’s get to our working example. Remember earlier I said Infrastructure-as-a-Service Clouds let you treat your servers like cattle. Well in sticking to the metaphor we need to first gain access to our cattle farm.

I’m going to assume you have an existing Oracle Cloud account or have signed up for a trial and you have the opc CLI installed.

Since we might have multiple accounts for our organisation, maybe different accounts per department or purpose (e.g. development, production etc), we may need to have multiple profiles. In our example, we are only interested in one: the cattle farm. Let’s create a folder for storing our profiles and another for storing our credentials.

mkdir -p ~/.opc/profiles
chmod 700 ~/.opc/profiles
mkdir -p ~/.opc/credentials
chmod 700 ~/.opc/credentials

Now let’s create a file for storing the password used by our profile at ~/.opc/credentials/cattle-farm.

touch ~/.opc/credentials/cattle-farm
chmod 600 ~/.opc/credentials/cattle-farm

Next, we create the profile itself at ~/.opc/profiles/cattle-farm with the following contents:

{
  "global": {
    "format": "text",
    "debug-requests": false
  },
  "compute": {
    "user": "/Compute-604021468/admin@cattlefarm.example.com",
    "password-file": "~/.opc/credentials/cattle-farm",
    "endpoint": "compute.aucom-east-1.oraclecloud.com"
  }
}

The user is the two-part identity: your identity domain (the Compute-604021468 part) plus the username you log in with. The endpoint is the REST endpoint for the compute site your account lives in, which you can find on the service details page of your Oracle Cloud dashboard. With the profile in place, every opc command can target the farm with -p cattle-farm. Now we are ready.

Defining our “image”

This is where the baking happens. Packer reads a JSON template, spins up a temporary builder instance from a base image, runs our provisioning scripts on it, snapshots the result as a new machine image, and destroys the builder. The recipe is three small files.

credentials.json holds the non-secret connection details:

{
  "username": "provisioning.automation",
  "identity_domain": "602302376",
  "endpoint": "https://compute.aucom-east-1.oraclecloud.com/"
}

secrets.json holds the password and stays out of source control:

{
  "password": "XXXXX"
}

And cow.json is the recipe itself:

{
  "builders": [
    {
      "type": "oracle-classic",
      "username": "{{user `username`}}",
      "password": "{{user `password`}}",
      "identity_domain": "{{user `identity_domain`}}",
      "api_endpoint": "{{user `endpoint`}}",
      "source_image_list": "/oracle/public/OL_7.2_UEKR4_x86_64",
      "shape": "oc3",
      "image_name": "OEL_7.2_UEKR4_Base_{{isotime \"20060102-150405\"}}",
      "attributes": "{\"userdata\": {\"pre-bootstrap\": {\"script\": [\"...\"]}}}",
      "dest_image_list": "OEL_7.2_UEKR4_Base",
      "image_description": "OEL 7 UEK4 base image for general servers"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "script": "scripts/base-packages.sh",
      "pause_before": "5s"
    },
    {
      "type": "shell",
      "script": "scripts/flush-disk-cache.sh",
      "pause_before": "5s"
    }
  ]
}

A few things worth noting:

One command runs the whole bakery:

packer build -var-file=credentials.json -var-file=secrets.json cow.json

Releasing the cattle!

With a baked image in our image list, launching cattle is a data-driven operation. We describe the instances we want in a launch plan, cow-instance.json:

{
  "instances": [{
      "shape": "oc3",
      "sshkeys": ["/Compute-602302376/farmer@cattlefarm.example.com/farm-key"],
      "name": "Compute-602302376/farmer@cattlefarm.example.com/cow-instance",
      "label": "cow-instance",
      "imagelist": "/Compute-602302376/farmer@cattlefarm.example.com/OEL_7.2_UEKR4_Base",
      "networking": {
        "eth0": {
          "nat": "ippool:/oracle/public/ippool"
        }
      },
      "storage_attachments": [{
        "volume": "/Compute-602302376/farmer@cattlefarm.example.com/master-storage",
        "index": 1
      }],
      "boot_order": [ 1 ]
  }]
}

And release it to the farm with:

opc -p cattle-farm compute launch-plan add cow-instance.json

Here is the part that matters: run that command, terminate the instance, and run it again. Same result. Run it five times behind a load balancer, same result five times. No SSH session ever needs to touch these instances, because everything they are was decided at bake time. When one misbehaves, it’s taken out back, shot, and replaced on the line, exactly as the metaphor promised, and nobody sheds a tear.

Where to from here

We now have the two raw ingredients of Immutable Infrastructure on Oracle Public Cloud: a bakery (Packer templates under source control) and a herd (launch plans stamping out disposable instances from baked images). The natural next step is to describe the surrounding infrastructure (networks, storage, load balancers and the instances themselves) declaratively with Terraform, so that the whole farm, not just the cattle, can be rebuilt on demand. Terraform’s provider ecosystem includes Oracle Cloud, and it pairs beautifully with the baked-image approach shown here.

References

← All posts