HubSpot transformed their aging, monolithic SaaS application into a vibrant matrix of 200 services, and they transformed their stressful release cycles into a stream of 100 change releases per day. Along the way, they have grown from a promising startup to fast-growing juggernaut. A few years ago I had dinner with both employees, founders Dharmesh Shah and Brian Halligan. On a recent visit, I found 600 people in their office.
HubSpot popularized the idea of using content and shared expertise to bring prospects to a Web site, eager to learn and to buy. They contrast this "inbound marketing" with the old outbound marketing, which sends messages out to unwilling victims, and is mostly ignored. Their SaaS product provides a complete suite of tools for small and medium-sized businesses to blog, publish, optimize, inform, and collect leads. They have used inbound marketing with some flair. The rapid growth of their SaaS service has been propelled by their Website Grader site, excellent blogging about online marketing, and other contributions.
I have used their blogging tool from its first launch, and I can testify that it's a useful service. However, in those early days, their marketing and ideas were noticeably better than the product. Even after four years of development, the software lacked some simple features like a reliable "remember me" login, and it was looking old. They decided to rebuild it with a new service architecture and new development teams.
Building all new code to upgrade an existing product is a famously risky maneuver. I once lost $6 million on such a bet. However, the new HubSpot product quickly surpassed the old product, and is now adding features and invading related categories at a rate that makes it a serious threat to competing vendors. This transformation is impressive because of its speed - it took about one year - and the fact that they were able to do it incrementally by ramping up new services and steadily ramping down the effort on the old codebase.
I interviewed Elias Torres to dive into their development process. I found him in HubSpot's bustling headquarters, where small teams worked under monitors showing real time performance and usage statistics.
HubSpot runs a pure form of MAXOS with programmer-driven service teams.
HubSpot teams have a tech lead and two engineers. There are about 20 teams maintaining about 200 services, so each team has about 10 services. There are about 30 other members of the development team doing design and product management. The small teams are organized into "clusters" which support complete products, or categories like mobile. Each cluster has a UX designer and a product manager.
To change a service, a developer submits a merge request. A colleague reviews it and merges it into the Git master branch for the service. A QA system automatically pulls the current master branch for each service and runs automated tests. Currently, the QA system is testing code from 500 different repositories that contain code and configurations. Integration tests include Selenium functional tests, and tests are added for bugs that customers have found. If the integration tests look good, the developer runs a command to deploy the modified service to production servers.
In order to make sure that every service gets expert attention through its complete life cycle, HubSpot gives developers a lot of responsibility. HubSpot development teams are responsible for design, programming, testing, release, monitoring, and responding to production problems. They can call on their cluster UX experts for design. They have no full time QA professionals. They have a custom monitoring and alert system that alerts members of a specific development team if there is a problem.
HubSpot teams are in many ways self-organizing, because they get so much monitoring and information about the services they own. The can see load, speed, and errors on a real time dashboard. They read customer questions and complaints submitted to the HubSpot knowledge base. These various inputs are combined into a "quality score." If they release a lot of new changes, the quality score is likely to decline. When it goes below a threshold, the team focuses its effort on improving quality. So, the process balances itself.
Elias says "it's all about empowering the team to do as much as possible, as quickly as possible. Tools provide guard rails." HubSpot operations provides their service teams with proprietary tools for configuration, release, and monitoring. All of their services and servers (2000 servers on Amazon) are mapped in a dashboard with red/yellow/green indicators.
They have sophisticated tooling for feature switches, which they call "gates." There is a centralized gate manager with a Web based editor. Developers and product managers can turn on features for a complete system (QA or production) or for individual customers. Gates allow them to do usability testing with selected customers before they unveil.