Google runs an extreme form of centralized continuous delivery. According to a report at an event in 2013, they have 15,000 developers working on 5,000 projects, committing to ONE current version of the code. Their single source code tree contains 100M lines of code. They run continuous tests on the "head," or most recent version, and when it looks good, they release it.

Their technique is successful. Google runs many of the Web's biggest services, and runs them reliably. Their products adapt quickly, and in some cases they can go from idea to release in 48 hours.

Google's approach is in large part determined by the way that they share code and Web services between projects. An engineer reports that developers "can make changes to kernel, Gmail, and other Google apps in the same changelist." This ability to make changes across the entire system allows engineers to make structural changes without a lot of organizational work and delay. However, it also means that they must run a centralized process with a single version of the test system. They can't run test systems for one component. They need to be able to test every service for any changelist. The Google test system is continuously compiling, caching and deploying all of the components required to run a test of the most recent code version - while running 100M test cases per day.

Many of the other aspects of development at Google fall out naturally from the decision to build and test one integrated system.

They require code review. If you are running test and deploy from one shared version of the code, you want a review before the code is committed.

They typically work in small, co-located teams. Google is a sprawling business, but most of the programming work is done by groups of 3-4 people who work together in one office. This makes it easy to get a code review. Each team has a tech lead. They don't use non-technical project managers and Agile functionaries. When your teams rely on code review and code deploy, you use tech leads who speak code.

They have a "test engineering" group to build and run the testing systems. The test engineering group is about 15% of the total developer base, which seems quite efficient to me.

The test engineers think a lot about "social engineering": creating a social framework that motivates developers to fix bugs and write tests for good test coverage. They keep leaderboards that report on bug counts and test coverage by developer.

Google uses a code management system that is based on Perforce. Perforce handles very large code repositories containing multiple projects. It also has a structure called a "changelist" that holds changes which are ready for review.

Scaling the organization with services, and continuous integration

Google was able to grow to a huge scale and an extremely complex environment because:

• They divided their vast Internet system into thousands of "services."
• Developers make frequent changes to services. However, before they release a change, they test it with the most recent version of all the other services. If a test script finds problems in the interactions between two services, it can tell developers who to contact to resolve the problems.

Google is using continuous integration to replace other, more complicated forms of planning. For decades, large-scale IT projects crashed and burned because project planners could not figure out all of the dependencies. Google can manage at an even bigger scale because they replaced this project management with a testing machine that continuously finds dependency problems. It is a fantastic management innovation, which I will describe in the chapter on "Continuous Enterprise."