"How do I test it?" is the fundamental question in any
continuous release or Kanban process. We make batch releases so
that we can test everything in the release. If we take away the
release testing phase, we need some special magic to make sure we
aren't releasing bugs.
Actually, we don't use magic. We use test layering. Any change
passes through a series of tests. Together, these tests filter out
most of the problems.
You can use this layering idea to increase quality, or
increase speed. The continuous process doesn't inherently
release more bugs than a batch testing process. It gives you a
choice.
- You can add more layers or more tests to increase quality. You
would do this if you see a quality problem, or if your product has
more and more users who rely on it. For example, we found that when
we moved to continuous delivery, we held features longer in the
"switch new features" layer, so that we could improve design and
usability.
- You can remove layers to increase speed. Use this technique to
accelerate an early stage project.
You will want to add testing where it is most efficient. When
adding tests, many people will start at the top of the stack above,
by monitoring a production release, or at least collecting bug
reports. That's a type of test. They may add manual or automated
regression testing to that so that they find problems before
customers find them. Then they will add unit tests, so that anyone
working on code can check it. In practice, you will add layers
going backward from the production release in order to find errors
closer and closer to the time when the code changed.
Any individual change runs through these layers from the bottom
to the top.
Let's look at some typical test layers, in the order that a
developer will use them to test a change.
- If your developers are writing unit test scripts, they will run
the test scripts locally to make sure that they have not broken
anything obvious. When working on a big system, programmers will
select a subset of tests to run.
- After the programmer submits code to the VCS, a continuous
integration system can pick it up and run a bigger set of automated
test scripts. It's useful to run these tests before code review, so
that people don't waste time reviewing changes that failed
tests.
- Code review makes the whole process work. The reviewer will do
a manual test, and will also look for standards compliance and
automated test coverage. The reviewer can ask for automated tests
if they are missing. If you want your programmers to write
automated tests, you will probably need to require code review. If
you require code review, you have a good chance of getting code
that meets standards for style, architecture, and test
coverage.
- Changes will go to a system where human testers can look at
them and run more extensive tests. Story owners, tech leads, and QA
professionals will look at the change. QA professionals are
typically called on when developers need them in a consulting
role.
- New features will be released, but hidden with feature
switches. New features may go through an extended period where they
are switched on selectively for beta testing, usability testing, or
A/B testing.
- The released software is monitored for errors, changes in
usage, bug reports, and other user feedback.