Assembla uses a distributed continuous delivery system in which developers use multiple code branches and test systems. This is a common configuration for Web and SaaS shops, because it allows them to release every change directly to production servers.

Distributed teams. Assembla also uses fully distributed teams, with 25 people spread across 10 different countries. This is also a common configuration for modern capital efficient startups. We recruit wherever we can find the best people. We use two-week trials where we throw the new candidate in with an experienced tech lead. Developers have organized themselves into 3-person feature teams, which improves online collaboration, and gives them two buddies to to ask for code review when they need it. Each team has a tech lead who can call on a specialist for graphic design or QA.

Kanban task management. We have found that development teams can work effectively off an online Cardwall. They pull their own work. We do frequent reviews of the amount of work in progress to keep it manageable.

Automated testing with on-demand test systems. Over the course of a year we invested in test infrastructure. We currently use cloud computing to start complete test instances of our system for every merge request. This allows us to run automated tests and to hand off a working system to a QA person and story owner for manual testing and review.

Story-owner product management and user experience design. One person is responsible for constructing the story, for design, for "unveiling" the feature, and for follow-up and usage statistics.

Git-based code management with short-lived feature branches and required code review. We use the distributed continuous delivery pattern (described in this guide), which allows us to release each change after it has been tested in its own test environment. We started with feature bigger teams that shared a development branch and a test server. However, we found that a problem with one change would block other changes from being released. Changes would accumulate for several days, and snowball into large releases that were difficult to debug. So we looked for ways to move each change through the system independently.

CTO Michael Chletsos explains:

The real decision that we made was to make one developer responsible for a change, from development, through testing, to production monitoring. We went through a series of steps to "unblock" developers:

• We added on-off switches to significant changes, so that developers can release them without being blocked by story owners. Story owners can control the "unveil" of a feature after it is on a production system.
• We provided on-demand test systems. Each change gets its own test system that runs automated tests, and also shows the change to story owners and QA. At Assembla, which is a small company, we use the magic of cloud computing to run up to 60 test servers at one time.
• Code review is required. This creates a blocking step, but we need it because it gives us a place to ask for test scripts that feed the automated test process. Manual QA is not required, but we put in some features to make it easier. For example, when a merge request gets deployed to an on-demand test server, we put a clickable link to that server on the related feature ticket.
• Developers trigger the deployment of a change. Developers can deploy whenever they want. They tell a bot to deploy a merge request.
• The Master version is always deployable because it is always deployed. The bot merges to master, runs an automated smoke test, and deploys. Test must take less than five minutes.
• We opened up monitoring so that developers can see all of the details of load and usage on the production servers.

Technical details: Story of a code change at Assembla

Here is the story of how a code change is handled at Assembla:

1. Dev commits a code change

2. The code commit is referenced on a ticket

3. A Merge Request (MR) is created by the developer (this could be a push to the master with a protected branch)

4. Jenkins picks up the change based on a webhook fired when the merge request was created

5. Jenkins, developers, and QA do several things in parallel:

• A new staging environment is created with the current code plus the changeset, and the ticket is updated to notify QA that the staging environment is ready.
• All integration tests are performed on the current production release plus the changeset; when it is completed the merge request receives a vote of +1 or -1.
• QA tests the staging environment.
• Developers perform a code review.

6. When tests are completed and the merge request receives a vote of +1 from development, Jenkins and QA, the developer asks the bot to deploy the merge request in a chat:

• 6a. The bot merges the merge request to the master.
• 6b. The bot deploys the change out to production.
• 6c. The bot writes a link into the ticket that shows where the changeset is being deployed

7. The change is deployed to the production environment.