I'd like to thank our friends at Perforce for putting us in touch with Edmunds.com, one of their customers and an excellent example of a company that has used the principles of Continuous Agile to strengthen its competitive position.

Edmunds.com provides authentic automotive information and a high-quality online shopping experience to automotive consumers, delivering the scoop about shiny new automobiles to more than 18 million unique monthly visitors. They also drive hundreds of mobile apps through their external API. They stay in top gear by releasing several times per day, and by planning product initiatives on timelines as short as weeks.

According to Ajit Zadgaonkar, Senior Director of Software Engineering, It wasn't always this way. He remembers when the business was dependent on annual plans with a quarterly release schedule. MS Project plans caused them to "lock down" on finishing features that weren't needed. Engineers were frustrated because they had to wait 90 days or more to see their changes deployed. Sometimes their innovations never made it to release.

By moving to more frequent release schedules, the engineers can see successes every day, and quickly make adjustments when new features are not as popular as expected. Ajit says that the teams have lost their fear of failure. The business has more appetite for innovation and experimentation.

Edmunds illustrates how the themes of Continuous Agile link together, sometimes without conscious planning.

Incremental improvements

Edmunds improved their release frequency incrementally: from every 3 months, to every month, to several times a day. They consolidated all of their code changes onto one release branch, and added feature flags to hide the features that weren't ready. They re-engineered their 40 person traditional QA team into a software engineering team. They enhanced the traditional software developer role to a well-rounded "software Engineer" with software development, testing and "Operationability" responsibilities - the big difference being that quality and critical software engineering functions were built into the role rather than being bolted on. They added test layers to build confidence. They provided access and visibility into production monitoring dashboards.

"Results only" teams

Before they started releasing more frequently, Edmunds moved to a distributed team structure. The whole team officially works out of their office in lovely Santa Monica. However, they aren't required to be in the office at any specific times, and often they aren't. Ajit says that, although it took two years to implement the new "results only work environment" across the entire company, it substantially improved both happiness and velocity.

Service architecture

Moving to a service architecture was a key step. Edmunds broke their monolithic Java app into 36 different services. Each service can be tested and deployed separately. A development team is typically totally responsible for one front-end service or feature, and has shared responsibility for all of the back-end services that it consumes.

Like HubSpot, Edmunds is using the "matrix of services" architecture and process. Each service team makes changes whenever they are ready, with an understanding that their changes will be backwards compatible, and will not break other services. Each service goes through centralized compatibility tests and regression tests to ensure it works with the most recent development version of the other services.

The promotion process uses a centralized continuous delivery approach. Programmers commit code to one Perforce Versioning Engine branch. The most recent code is compiled into a Java file and passed into a Nexus repository for testing and promotion. A stable service gets promoted into the "blue" bank of servers and switched on, and then the development team monitors the results.

Test layering

The Edmunds release process is a beautiful example of test layering. In this diagram they show how a change goes through eight layers of automated testing.

Edmunds doesn't worry about having high "test coverage" at every layer. Instead, when they see a problem, they write a test to prevent it from happening again and select the best place to put the test.

Edmunds uses the bank-switching release method for their Web site. At any given time, they have a "green" active bank and a "blue" inactive bank. After they update and test the blue bank, they switch over to the new code on the blue bank. If they see any problems, they can switch back.

Ajit emphasizes that it is important to provide developers with fast testing and feedback so they can address problems immediately. All of their automated test processes are optimized and parallelized across multiple machines. Final production tests must run in less than 8 minutes. This process requires a lot of servers. They run their test layering with a development bank, a staging bank, and additional servers for build, test, and monitoring. To handle all of this they are currently using a hybrid public and private cloud architecture.

Data-driven product management

Edmunds team members can get the data they need to make good decisions. Everyone in the company has access to key success factors, A/B test results, and the health of various environments and features. Ajit suspects this may be just the start. As with many organizations, the next step for Edmunds is to use real time data streams of continuous feedback to help them identify areas of improvement, and to channel energy to focus on high impact issues.