Draft of Codenvy story, from Tyler Jewell, CEO of Codenvy
Codenvy was faced with a huge challenge in 2012. We needed to go from one release per month to 4 releases, while also shipping 4 products on the same code base, both cloud software and downloadable installation software. We had to do this while increasing our team size 400% and support all of our legacy architecture. If we didn't, we'd go out of business…
Codenvy provides SaaS developer environments with offerings for public cloud at Codenvy.com, private cloud for on premises enterprise installations, and ISV solutions including custom tooling, APIs and plug-ins. Codenvy's public cloud is used by over 100,000 developers. SaaS developer environments improve velocity of control for software teams by eliminating inefficiencies and constraints in the construction process of software. This includes eliminating days lost configuring environments and hours spent updating software and plug-ins. The Codenvy cloud is integrated with a variety of partners that are constantly changing their interfaces and modes of connection.
Codenvy originally started out as a pure public cloud with monthly releases. These releases included both feature/functionality along with patches to existing issues. As the company grew, the mandate became to more aggressively pursue relationships with enterprise organizations. Enterprises had strict compliance, auditing and security requirements which dictated the need for an on premises SaaS offering that they could orchestrate with their own developer community. Codenvy quickly needed to become a multi-product company, making cloud-speed iterative improvements to Codenvy.com but also keeping those same changes as part of an on premises install enterprise cloud.
Codenvy also expanded the R&D team from 4 people up to 28. The growth in team members meant that the system needed to be divided into horizontal, but compatible tiers that could each release on independent cycles.
Codenvy needed a continuous delivery process that achieved a number of goals: a) allowed for a fracturing of the platform until multiple, co-dependent tiers that could be independently deployed, b) the ability to release frequently in an entirely automated fashion to a variety of environments including acceptance, staging, cloud pre-production, and production, c) the simultaneous release of public cloud and three downloadable versions of Codenvy: SDK, trial single server installation of Codenvy Enterprise, and full installation of Codenvy Enterprise, d) the matched release cycle of documentation, marketing, and sales workflows that supported the IP that was being pushed.
As a first step, the R&D organization committed to weekly sprints following a basic agile model. Clear epics and testable release assets after each sprint were provided. The QA team was disbanded and they were transitioned into automation engineers that were part of their component team. QA engineers were responsible for automation of the builds and relevant assets.
As a second step, a devops team was formed to create a release system that would push changes into all production environments simultaneously based upon tiers of quality threshold attainment. Engineers, writing in Java (and also using Codenvy to write their code to create Codenvy!), would perform pre-flight quality checks prior to check-in. A pre-flight check was a partial run of certain continuous integration code analysis and unit tests that would give input prior to check-in. After check-in, a Jenkins system was installed to orchestrate the packaging of the code, and then pushing that code into acceptance, staging, and pre-production. Acceptance servers were used for the coordination of discussion between product managers and engineers, staging environments are used for simultaneous scale testing, and the pre-production environment was suitable for marketing / documentation teams to prepare launch-relevant assets against that code contribution. For reporting, commit messages and relevant content became ubiquitous and available to everyone in the firm, so they could track updates to various environments and be triggered to perform manual actions necessary to support that ultimate release. The push to production was then coordinated by devops against scheduled maintenance windows.
As a third step, the Codenvy team broke into horizontal tiers that would independently deploy and release. These tiers became analytics for data crunching and dashboarding, cloud infrastructure for multi-tenancy and security, plug-in / SDK tier for structuring the format of a SaaS environment, IDE team, plug-in development team, and devops, which included system administration, operations, documentation, and automation. Each engineering team became fully self-sufficient and contained engineers that worked on QA, UI, and back-end systems, since each horizontal tier had some exposed UI and / or API. All of the tiers were mandated to be forward and backwards compatible and the teams became responsible for compatibility testing as part of a delivery release & build cycle. Now, Codenvy is able to release APIs, new thin clients, CLIs, and various infrastructure improvements independently and concurrently.
As the final step, to target their enterprise accounts, Codenvy needed to take a single code base and package it for different audiences. For developers, their online Codenvy.com was the primary portal. For software vendors creating extensions, they packaged a single server, single tenancy SDK that also has its own shadow public cloud at sdk.codenvy.com. And for enterprises who wanted to run their own full development cloud, Codenvy needed to ship a single server trial installation of Codenvy Enterprise while also making a multi-server full installation available as well.
With Codenvy releasing weekly after one year (and steadily increasing release rates), the potential to have too many versions of the SDK and Codenvy Enterprise littered throughout the world would have created a difficult customer service and support situation. The decision was that we needed to extend continuous delivery all the way out to their own customers. Each Codenvy IP component installed needed to release as frequently as the Codenvy core system. To support this, Codenvy implemented an installation mechanism using Puppet and Puppet Master. All installations of the SDK or full installation of Codenvy Enterprise would connect to the Puppet Master, and synchronize their installations every couple hours. Codenvy engineers had to build an incremental update mechanism to keep the system alive 24/7 and also deploy improvements to on premises installations without impacting any active developers. And for organizations that created firewall blocks, Codenvy offers a proxy connector to create a secure bridge to the Puppet Master. Finally, for the trial installation, it was bound to a 30 day license, so it is updated automatically on each push, replacing past binaries.
In 2012, Codenvy was able to do 8 releases with a small team of engineers for their public cloud only. In 2013, they did 24 releases. And in 2014, they are now shipping four products simultaneously across 5 acceptance servers, staging, pre-production, and two production environments. For 2014, the continuous delivery objectives are to get to a point of fully automated release criteria with production pushes that do not require human authorization, and complete rollbacks in case of issues.