Cox Media GroupKryptonite
Safely deploying on the cutting edge
This talk will cover how we do deployments. It is the first deployment system that I've been happy with, which says a lot. It is a system that we have been evolving over the last 6 months or so, and has made our developers lives a lot easier. There are a few different points that make it successful, as well as useful.
Interestingly, there is very little Django specific about this setup. It is how we deploy Django applications, but we also deploy flask and standard WSGI applications using the same framework. A slightly modified version of the code is also used to deploy our Java applications, so the tooling is genericly useful for more than just Django programmers
The talk will have 4 parts, that will cover the following:
- The process around Version Control that we use
- The tooling to deploy the code
- The environment that the code gets deployed in to
- The process of deploying the code across all of our servers, a subset at a time
The first part is that we manage our source code in a specific fashion. It's a pretty standard git branching model, where we have a master branch that is the latest stable code. All on-going work is done on feature branches. Then we have a production branch that is what is currently in production. Each push to production is tagged with the date. Git allows us to keep a record of the history of development in one place.
Next, to actually handle the deployment of the code to the servers, we're using Fabric. To show the full power of the system, we might normally run a command like fab host:web-0 deploy:airship restart:airship monitor:airship. This is mostly abstracted out from people in the day to day use of the system. The deployment system has the following features:
- Updating source code checkouts
- Making VCS tags
- Deploying code
- Migrating the database
- Updating media files
- Restarting services
- Rolling back to the previous version
- Allowing the person deploying the ability to easier monitor logs/status of services
- One command "smart deploys"
The python environment that things are deployed into is a fully standardized setup using supervisord and gunicorn, along with a number of other helper utilities. It integrates into the init.d system on the host, so that normal management commands work. Each service that we deploy has it's own environment, which I'll go into more depth on.
Finally, We are also working on a system that will allow us to automatically health check deployments and automatically rollback the deployment if we are seeing elevated error rates. We handle deploying the code to a "canary" system first, then roll it out to half our servers, then the other half. This system is implemented currently at the prototype level, but should be in production use before the talk is given.