Image: “Delivery Dog” by AnimalCrew
We’ve been doing Continuous Delivery for a while at Redbubble, and we’ve learned quite a bit, so I thought I’d share a few questions you should ask when making the transition from Continuous Integration to Continuous Delivery. Hopefully our experience can help you, whatever stage you’re at.
But first, a quick refresher on what exactly Continuous Delivery is. When we are doing Continuous Delivery, we are doing Continuous Integration by definition. In addition to having an automated build, Continuous Delivery requires that any passing build can be deployed at any time, at the push of a button. This is more advanced than Continuous Integration, in which we always know whether our software builds correctly or not. It is less advanced than Continuous Deployment, which requires not only that every build be potentially deployed, but that it be actually deployed.
Martin Fowler wrote an excellent description of Continuous Delivery if you’d like to know more.
One of the keys to a successful Continuous Delivery process is that deploys should be a fuss-free affair – reliable and preferably frequent. After all, there’s no better way to prove that your code is ready for deployment than to actually deploy it.
If fuss-free deployments are the key to successful Continuous Delivery, then reliability is one of the keys to fuss-free deployments. If you can push the envelope further and allow (or encourage, or even expect!) your teams to deploy their changes frequently, you will demonstrate the reliability of your application, your deployment process and your team. You’ll also be showing a huge amount of trust in your people. This, along with lots of other important things, can help to foster a culture of trust and responsibility within your team.
For the purposes of this post, let’s assume that you have some code that has passed your Continuous Integration process and is ready to deploy to production. Here are a few questions to help you think about what you might need to change or improve as part of a transition to Continuous Delivery.
How easy is your deployment process?
If your deployment process is easy to use, chances are it’s pretty reliable.
Another way to ask this question might be: How many steps do humans need to perform in order to get the code into production? The more steps humans need to do, the greater the chances that human error will creep in and mess up your deployments.
If the answer to this question bothers you, consider investing in automation. If your process is already automated, ask yourself why there are so many steps. Could you add a layer of indirection to run them in sequence, without human intervention?
What percentage of attempted deploys are successful?
This is a good one to consider as a measure of the reliability of your process. Take an initial measurement without judgement, and measure again regularly to gauge your progress.
How often do you deploy?
This is another good number to watch, and one that non-technical people can understand reasonably easily.
There’s also a process improvement side to setting a target for deployment frequency. The more often you deploy, the more pain your team will feel if deployment is painful. Notice the pain points, then fix their root causes, and celebrate when things get better!
Who knows how to deploy?
Generally, the more people you have who know how to deploy, the more usable your process can be said to be.
Is there a single person or team in charge of deploying the code? If there is, it may be because your deployment process is hard to understand.
Look for opportunities to make the process more comprehensible. You may be able to achieve this through documentation. You could choose to solve it by refactoring (e.g. separating a “God script” into shorter scripts for discrete sections of the process).
It may be worth your while to take some extra time and overhaul your process. Examine each step and ask “What problem caused this step to be introduced? Is it still a problem? Can we solve it outside the deployment process?”
If any developer/engineer/coder can start a deploy with confidence, that’s a good sign that Continuous Delivery might be within your technical reach.
Who knows the deployment process intimately?
If the set of people who know the deployment process in detail is the same as the set of people who can deploy at all, then watch out! This pattern suggests that you need detailed knowledge in order to deploy, which in turn implies that there are lots of places in your process where things can go wrong.
If that’s the case, there’s probably a lot of work ahead before you’re ready for Continuous Delivery. Work on error-proofing your process, either checking that conditions are right before the process goes ahead, or changing the process so that it works (or fails safely) under any conditions.
If you’re aiming to eventually adopt Continuous Deployment, where every merge to master/trunk is released automatically, then you’ll need your process to work reliably in the vast majority of cases, and fail safely in the rest.
Is special equipment required (e.g. a particular machine)?
Perhaps your process involves a blessed “deployment machine”. There are a few reasons you might need one. Among others:
- The machine may hold special credentials or network access required for deployment.
- The machine might be the place where the code for executing deployments is stored.
In either case, that machine is critical to your team’s operation. Make sure you’ve planned how you will recover if that machine goes down. First steps might include making sure that your deployment code and credentials are backed up safely, and that your deployment code is under version control. If your deployment machine has some kind of special network access, consider how you’ll access the network if that machine dies.
Are there gates such as human approval steps?
Human approval steps can be tricky to negotiate in an automated deployment process. They often mean interruptions and delays, especially when the decision maker gets sick or is unavailable. Consult with decision makers about the process they follow to reach a decision, and consider whether you could automate that process, for example by adding tests to your Continuous Integration build.
Which services have to work correctly for a successful deployment?
Do you depend on version control? A CI system? Artifact storage? Whatever those services are, they need to be reliable before your deployments can be reliable or frequent. And you want frequent, reliable deployments, because the more frequently you deploy, the faster you can release new features or fix problems.
Hopefully these questions help you think about where to focus your effort in preparing your team to adopt a Continuous Delivery process. Are there more useful questions I haven’t thought of? Let us know in the comments!