Release Strategies

My wife often calls me to announce that a major bank or airline web site she is using has gone down. Often they go down for weekend releases. She thinks their problems will make me feel better about the downtime that we have at Assembla, which is about one hour per quarter, or about one hour per 300 releases. But I think that tolerating any downtime is wimpy. There are a number of release strategies that those guys could use to avoid it. If you want to release more frequently, you have to be able to release without disrupting your users.

Beta releases

You can lower the risks of your continuous delivery process if you release to a subset of users and let them provide usability feedback and help you find bugs. These are your beta users. Most software companies inflict the beta version on their internal users. You should always be looking for ways to divide up your external user base and find beta users. You may want to test with free users. Some of your users will volunteer because they are innovators.

If you distributed installed software - including mobile clients - you will need beta users who will accept and download the beta releases.

  • Automatically notify beta users when a new release is ready
  • Send them release notes. You can automate the production of release notes by reading them from your development ticketing system. That will encourage users to try the new releases. You get extra credit for giving them a place to click to provide feedback.

I often tell my teams that we WILL release, but they can decide who to release to. If you partition your user base and add feature switches, you will have a lot of options and continuous delivery will not be stressful.

Automatic updates

Sometimes you can just run a script to update your software on customers' sites, if the software is designed to retrieve updates. Most consumer software works this way.

Sequential deployment to servers

High-volume online services can just update their banks of servers, one server at a time. That is what we do. Be sure that you can easily roll back.

Swap banks

If you have a complicated cluster, you can't update one server at a time. However, you can swap a whole cluster in seconds. Set up two banks of production servers. Run one in production. Deploy your new release to the second bank, and test it. Swap. Don't be a wimp about building new banks. It's cheaper to buy more hardware than to keep people waiting.

Swapping banks is the first thing I try with new customers that have complicated systems.

Service architecture

If you have a very complicated system you may be forced into a service architecture where you have many servers that call each other. Use the techniques described here to update only one component at a time. Even if your services are well tested, you will sometimes release something that causes an error. However, if you update one component at a time you will often find that the error only degrades a small number of features, while the rest of the system works fine. You can even program your service users to handle changes and errors.