Test Layering

"How do I test it?" is the fundamental question in any continuous release or Kanban process. We make batch releases so that we can test everything in the release. If we take away the release testing phase, we need some special magic to make sure we aren't releasing bugs.

Actually, we don't use magic. We use test layering. Any change passes through a series of tests. Together, these tests filter out most of the problems.

You can use this layering idea to increase quality, or increase speed. The continuous process doesn't inherently release more bugs than a batch testing process. It gives you a choice.

  • You can add more layers or more tests to increase quality. You would do this if you see a quality problem, or if your product has more and more users who rely on it. For example, we found that when we moved to continuous delivery, we held features longer in the "switch new features" layer, so that we could improve design and usability.
  • You can remove layers to increase speed. Use this technique to accelerate an early stage project.

You will want to add testing where it is most efficient. When adding tests, many people will start at the top of the stack above, by monitoring a production release, or at least collecting bug reports. That's a type of test. They may add manual or automated regression testing to that so that they find problems before customers find them. Then they will add unit tests, so that anyone working on code can check it. In practice, you will add layers going backward from the production release in order to find errors closer and closer to the time when the code changed.

Any individual change runs through these layers from the bottom to the top.

Let's look at some typical test layers, in the order that a developer will use them to test a change.

  1. If your developers are writing unit test scripts, they will run the test scripts locally to make sure that they have not broken anything obvious. When working on a big system, programmers will select a subset of tests to run.
  2. After the programmer submits code to the VCS, a continuous integration system can pick it up and run a bigger set of automated test scripts. It's useful to run these tests before code review, so that people don't waste time reviewing changes that failed tests.
  3. Code review makes the whole process work. The reviewer will do a manual test, and will also look for standards compliance and automated test coverage. The reviewer can ask for automated tests if they are missing. If you want your programmers to write automated tests, you will probably need to require code review. If you require code review, you have a good chance of getting code that meets standards for style, architecture, and test coverage.
  4. Changes will go to a system where human testers can look at them and run more extensive tests. Story owners, tech leads, and QA professionals will look at the change. QA professionals are typically called on when developers need them in a consulting role.
  5. New features will be released, but hidden with feature switches. New features may go through an extended period where they are switched on selectively for beta testing, usability testing, or A/B testing.
  6. The released software is monitored for errors, changes in usage, bug reports, and other user feedback.