If you have more than 20 items on the backlog, you will find it very hard to sort them by priority. You will naturally start to divide the backlog into topics, or into an outline-style project plan. An excellent way to handle this is to treat each important topic as an "epic" - a list of stories to implement a major deliverable.
Epic owners will select the high value deliverables to become "work in progress." They incrementally pull stories out of an epic, one at a time, and move them to your sprint backlog or work in progress.
A story is something that you can release at one time. You can put a story on the backlog, pull it for implementation, release it, and close it. An epic cannot be released at one time, because it contains multiple stories that are released at different times.
You will find it annoying if you put epics into the backlog list, or on the cardwall for current work in progress. They will hang there for a long time and always look stuck. In a lean, continuous system, you do not ever want to be stuck. Epics should go into a different view in your planning system.
You will want to divide your epics into small stories and build them incrementally. This will reduce stress, because you can use continuous delivery. It will reduce bugs, because it is easier to find problems in a small change with fewer lines of code. It will force you to identify the highest value deliverables, so that you can work on them first.
For example, my top level job today is Assembla content marketing. One of my major deliverables is an eBook about continuous delivery. I will build this incrementally, with help from a Web designer and my friends. I will show each chapter as it emerges. My planning hierarchy would look something like this:
Assembla content marketing (Top level stakeholder)
You can organize the top level of your outline in various ways. If you are a professional services company, you will want to put major customer accounts at the top level. If you are an enterprise IT shop, you will put line of business leaders or other "stakeholders" at the top level. If you are a software product company, you will probably end up with an organization something like this:
Strategic product development (owned by the CEO and product managers)
Architecture and technical debt (owned by the CTO)
Bugs (owned by customer support)
Marketing and funnel (owned by marketing)
These topics are not all the same type. Some of them are very specific changes, like bugs. Some of them are defined around a bigger issue or around a team. However, each top-level topic has someone who cares about it deeply. The support team needs to reduce bugs and support calls. The marketing team needs selling tools built into the product.
When you organize this way, each top level category, and each epic, will have an owner, someone who really cares about that epic.
Here is the trick that makes an enterprise planning process work.
Once you have organized your outline this way, you are ready to create your ordered backlog for the implementers. You don't have to order everything. You only need to select and order the next 20 items.
Since all of the epic owners care about what they are doing, it will be a big mess if you invite all of them to a meeting, let them lobby for their personal epics, and expect a product owner to select a small number of important items that will make everyone happy. Now it becomes clear why it is difficult to sort more than 20 items.
But decision-making is easy if we allocate a fixed number of stories to each epic owner. Let them choose X stories they can have in the backlog and in work in progress. In the example above, we might say that the CEO and new product team get 8 stories, the support team gets 4 stories, the marketing team gets 4 stories, and the developers and architects get 4 stories.
You can do this allocation in a batch, where everyone picks their stories at the same time. Or you can do it continuously, and allow epic owners to pick new stories when existing ones are finished.
This type of allocation is a central feature of enterprise planning frameworks like Scaled Agile Framework, Disciplined Agile Delivery, and Enterprise Agility. It is often called "Portfolio Allocation," because you are allocating capacity to a portfolio of projects.
Some team leaders will agree with Joel Spolsky that a software team backlog or cardwall should not be bigger than 20 items. This is a good way to maintain clarity for an implementation team. However, it is an extremely naive way to think about project planning. We send men to the moon and put three billion people onto a mobile network. We don't do those things 20 tasks at a time. The 20 task limit is not a natural limit for projects. It is a function of human psychology, the longest list that one human can reliably sort into priority order.
In contrast, with epic planning we can use many people to coordinate a more complicated project. With allocations, we can maintain clarity by sending stories to implementation teams in batches of 20 or fewer.