Some modern online services have hundreds or thousands of developers and release once, twice or 50 times per day. They do it with a distributed continuous process in which each change gets tested and released independently.

You cannot guarantee that your change will work with all the other changes produced by a big group of developers. To solve this problem, distributed continuous delivery leaps from "as early as possible" integration to "as late as possible" integration. With "as late as possible" integration you test each change with a recent production version of the code before you merge it with the other candidate changes.

Here is how it works:

1. As an individual or team committer, you put changes into your own branch or fork.
2. For testing, you merge the production version to your change branch. This gives you a release candidate on your workstation for testing. Git has a special operation called "rebase" that helps you merge the target version, and then batches your changes up into a single package for delivery.
3. Your development organization takes advantage of cloud computing to give every contributor an on-demand test environment. QA acts in a consulting role, looking at your test environment if you need help verifying that changes are good and desirable.
4. You do a final test and deploy. This is highly automated. You need to make sure that your change works with the most recent release, so you do a final update with automated tests. If this passes, the system will deploy.

Advantages

With the distributed process, you can release every change immediately. This provides a competitive advantage for online applications and service providers. The continuous release process also reduces stress by getting fixes out more quickly, and by allowing big changes to be completed in the minimum time needed to actually do the work. Finally, the distributed continuous delivery process can include a large number of contributors distributed in multiple time zones.

Disadvantages

With distributed continuous delivery, test environments are more complicated. Setting up multiple test environments creates a bigger workload for DevOps. Developers are responsible for maintaining versions and shepherding changes through tests. They need to know more about the test environments.

You may not have time to release all of your changes. It takes time to run the final test and deploy, because automated tests take time. Changes are tested and released sequentially. Tests for each change need to wait for tests to be completed on previous changes. If you have a lot of contributors, you may not have time to release all of their changes. To address this issue you can make a faster pre-release "smoke test," or you can cascade and batch up the changes.

Critics often observe that developers using this approach tend to work a long time on their branches and then cause integration problems when they merge big batches of code. Finding problems in these big batches of code is difficult. If you use distributed continuous delivery you should:

• Frequently "merge back" from the production version to your development version. If you do frequent integration and test in your development version, you will find and fix problems before you merge "forward" into the production version.
• Release changes as frequently as possible. When you release smaller amounts of code, you have fewer places to look for problems and you will find them faster.

Multiple test environments

You will want to have multiple test environments, so that you can test each change separately. Testers will act as consultants, helping when developers ask them to look at a specific test system. Building and maintaining these test environments takes some work. However, you can use the miracle of cloud computing and versioned configurations to deliver test environments on-demand, allowing you to work with an unlimited number of contributors.