Flexible Configuration Injection with StashBox

We recently implemented the AutoPilot Pattern using ContainerPilot and blogged about it. One of the significant issues that we dealt with in that blog entry, and in our implementation of the AutoPilot Pattern and our move to an entirely container-based solution hosted on Joyent Triton, was injecting configuration.

Injecting Configuration

By "configuration" we are referring to all of the static state required to launch and run your application. This includes traditional configuration settings and files, as well as things like SSL credentials (certs and keys), API tokens, SSH keys, etc.

The methods we have to inject configuration into a container include:

  • Environment variables (either baked-in to your image at build-time, or injected at run-time)
  • Files (baked-in to your imaged at build-time)
  • Volumes (mounted to your image at run-time)

There are pros and cons to each of these approaches. There are 12 Factor devotees who always want configuration in envronment variables. There are security conscious teams that want secrets built-in to the image to avoid leaking them to the environment and to gain file system security over the contents. And there are a lot of people who prefer to use volumes, in part because it's an easy way to inject whole files or sets of files into one or more containers at runtime.

As a solution provider ourselves, we wanted our solution to be non-opinionated about how configuration is injected, meaning that all of our containers needed to support each of these modes. And we needed the flexibility to allow our users to use our published public containers (meaning all configuration had to be able to be injected at run-time), or to bake-in all of the configuration into their own private/custom containers built with the same Dockerfile and support files.

But for our own deployment, we still weren't happy with the options that we had. All of our application configuration (both for Nginx and for our own server, Synchro) lived in a directory on Manta (our cloud storage). This included our SSL credentials, the basic config for Synchro, and a set of Synchro apps (whole Node.js micro-apps). Collecting the various configuration information for each container, formatting it so that it could be injected at run-time, and managing the resulting sets of env files, was pretty tedious and prone to error. And it gave us a copy of that configuration (spread across multiple env files) that we then had to manage separately from the underlying data, including keeping it in sync.

Why not just use volumes?

Different cloud providers have different constraints on how volumes can be used (including limiting the number of volumes, co-provisioning constraints, no support for host mounting, etc). And depending on what kind of volume you use, it may be difficult to reproduce that environment when not deployed on your cloud provider.

When we moved to Joyent and saw their constraints on volume usage, it became pretty clear that that was not a viable strategy.

Why not just use volume drivers?

Docker volume driver support is spotty (they don't exist for all platforms, and many are early stage). Relying on volume drivers also vastly complicates your development environment - in order to fully replicate your production environment the volume drivers have to a) be installed in the local development environment (can be challenging and difficult to maintain), and b) be configured to be able to reach their storage (which is not even possible with some volume drivers).

In our case, deploying on Joyent, this was not even an option as there is currently no Docker volume driver for Manta (though for the reasons above, we wouldn't have used it if it did exist).

So what did we want?

We wanted a non-opinionated solution to get configuration files from anywhere, at runtime, with no client tooling (volume driver, etc) and no micro-service tooling (cli tools, etc) required, and that would work in any orchestration/scheduling environment. Ideally, the containers getting the configuration would be completely abstracted from the source of the configuration. The solution should support the existing mechanisms of configuration injection (env vars, files, volumes) in addition to all major cloud storage providers.

And we realized that we had most of the pieces we needed to build such a solution laying around...

Introducing StashBox

StashBox is an http server/proxy that serves files from configurable sources, including environment vars, local files, http proxy, and cloud storage: Amazon (S3), Azure, Google, HP, IBM BlueMix, Joyent (Manta), OpenStack, and RackSpace.

Clients (typically Docker containers) can get files from StashBox without any tooling (beyond curl).

Usage: Deploy a StashBox container configured with one or more mount points to serve configuration files from the underlying source of your choice. At startup, each container that requires configuration files can simply curl their config files from StashBox:

curl stashbox/config/ssl.crt -s -S -f -o /etc/ssl/certs/mydomain.crt 

Here's a Picture:

The Project

StashBox is open source and the GitHub project can be found at: https://github.com/SynchroLabs/StashBox

To see StashBox in action (with ContainerPilot), check out the Synchro AutoPilot GitHub repo.

There is public image of StashBox on Docker Hub: synchro/stashbox

Configuring StashBox

StashBox mount points are defined in the StashBox config, and typically look something like this:

# StashBox config.json
{
  "mounts": 
  [ 
    { 
      "mount": "/stuff", 
      "provider": "file", 
      "basePath": "stash" 
    }, 
    { 
      "mount": "/config/ssl.crt",
      "provider": "env",
      "var": "SSL_CERTS_B64",
      "encoding": "base64"
    }
  ]
}

Note that StashBox itself can be configured via environment (run-time) or config file (build-time).

For detailed instructions on how to configure a mount-point for a specific driver, or on how to express complex StashBox configurations using environment variables, see the StashBox README.

Leveraging ContainerPilot

ContainerPilot is a great companion to StashBox. We can use the ContainerPilot preStart hook to process configuration, including potentially getting configuration from StashBox, before our application is launched (by ContainerPilot).

You certainly don't need to use ContainerPilot in order to use StashBox. You could alternatively launch a startup script from your container that did configuration processing and then launched your app (though you are replacing pid 1 in that scenario, and will have some responsibility for passing signals, among other things). ContainerPilot makes this a lot easier, and has many other benefits, so that's what we use.

In general, our configuration processing supports elements that have these properties:

  • The local file path may be specified via environment var, and has a reasonable default
  • The file contents may be injected via a base64 encoded environment var (if specified)
  • The file contents may be injected from a remote URL (if specified)

The preStart entrypoint for the Synchro Nginx image is shown below. Note that the URL environment vars are not specific to StashBox (though StashBox is what we will always us in our deployments).

Synchro Nginx preStart processing:

#!/bin/sh
# ContainerPilot preStart
#
preStart() 
{
  # Get SSL certs from env or remote URL...
  : ${SSL_CERTS_PATH:="/etc/ssl/certs/ssl.crt"}
  if [ -n "$SSL_CERTS_BASE64" ]; then
    echo $SSL_CERTS_BASE64 | base64 -d > $SSL_CERTS_PATH
  elif [ -n "$SSL_CERTS_URL" ]; then
    curl $SSL_CERTS_URL -s -S -f -o $SSL_CERTS_PATH
  fi;

  # >>> Process ssl.key, nginx.conf.ctmpl, etc as above

  consul-template \
    -once -consul consul:8500 \
    -template "/etc/containerpilot/nginx.conf.ctmpl:/etc/nginx/nginx.conf”
}

Note that adding config file URL (StashBox) support does not interfere with support for processing all of the other modalities of configuration injection. In the script above, the http server (StashBox) will only be contacted if the environment variable indicating the URL endpoint for the file is specified. So adding StashBox support does not require your users to use StashBox or prevent them from doing whatever else they want to do instead.

Deploying Synchro with StashBox

We deployed our own production Synchro API server (including web front-end, web-based mobile app services, and native mobile app API endpoints) on Joyent using StashBox to get all of our configuration from Joyent's Manta object store.

First, we pointed our Nginx and Synchro containers to StashBox for their config:

# nginx.env
SSL=1
SSL_CERTS_URL=stashbox/config/ssl.crt
SSL_KEY_URL=stashbox/config/ssl.key

# synchro.env
SYNCHRO_CONFIG_URL=stashbox/config/config.json

Then we pointed StashBox to where the config actually lives (in our case, Joyent Manta storage):

# stashbox.env
STASHBOX__mounts__0__mount=/config
STASHBOX__mounts__0__provider=manta
STASHBOX__mounts__0__basePath=~~/stor/api_synchro_io/config
STASHBOX__mounts__0__url=https://us-east.manta.joyent.com
STASHBOX__mounts__0__user=synchro
STASHBOX__mounts__0__keyId=<key id>
STASHBOX__mounts__0__key64=<base64 encoded key>

Neither the Nginx nor Synchro containers actually know where their configuration is coming from, and should we decide to change where StashBox is getting the configuration (to a local dev, test, or staging set of files, potentially even served from a different type of store), those containers do not need to know that, and their configurations don't have to change.

This allows us to use public, published containers, where all configuration is specified at run-time, and where the only thing we have to change to change the source of configuration files is the run-time StashBox configuration.

Here is what our deployment looks like using StashBox to get our config from Manta:

Synchro Deployed on Joyent Triton/Manta with StashBox

Benefits for our Project

  • We can get our configuration from Manta (there was no easy way to do that before).

  • We can consolidate all of our configuration in one place, which makes it easier to manage and easier to port.

  • We can run the same exact set of containers in any environment and be guaranteed the same configuration that we have in production (our staging test containers are using StashBox and accessing the same exact configuration on Manta as our production deployment - and those containers don't need to be running on Triton - they can be on a local development machine).

Conclusion

We built StashBox because we needed it and it fit our development and deployment model. If it's useful to you, feel free to use it. If you have issues or there are features that would make it work for you, let is know via GitHub issues.

Note that most of the cloud storage support is courtesy of pkgcloud, which we already used in Synchro (and of which I am a contributor).