When to Demo & When to Document (Two Approaches to Building Software)

Lots of product and engineering teams debate having “document-led” versus “demo-led” cultures of building.

These represent two opposite ways companies have for creating consensus and settling debates during planning and build cycles. A demo-first culture leads with tangible artifacts for people to react to and then iterates on that feedback. A document-first culture leads with documentation and analysis — whether a PRD, specs, PR/FAQ, RFC, or similar-style deliverable — to figure out work worth doing, define the scope, and set the intended outcomes.

The most visible implementations of these cultures have been at Amazon and Apple. As a quick summary:

Amazon is famous for their 6-page PR/FAQ documents that are aligned on by a leadership team before anything gets built. The bar for approval of a PR/FAQ is high, and the process becomes an effective focus-protecting/project-killing mechanism. (This is described in detail in “Working Backwards” by Colin Bryar and Bill Carr.)

Apple is famous for a culture of iterating through proof of concepts and hands-on demos, which are also reviewed by senior leadership. Each demo gathers crucial feedback that is used to continue problem-solving. Experience rather than business analysis is the crucial driver of decisions. Notably, these demos are started and iterated one with little-to-no original written specs and requirements. (This is described in detail in “Creative Selection” by Ken Kocienda.)

Most individuals I know have a strong preference for one culture or the other. Here’s a sample:

However, most teams I’ve seen or worked with try create a hybrid ‘best-of-both-worlds’ culture and implement it via a one-size-fits-all, we-all-use-the-same-process-across-all-projects approach. They end up with a combination of upfront documents (PRDs/briefs) and build-demo-build processes (sprint-style cycles of development). I've been guilty of this myself.

On the surface, it makes a lot of sense. Yet I’ve never heard anyone brag or speak positively about these hybrid approaches. Rather than getting the best of both cultures, they end up feeling like the worst: the documentation parts feel like bureaucratic pre-work and doesn’t validate ideas against customer needs (or get nearly enough projects killed), while sprint-style development gets to biweekly units of planning/work but mostly pays lip service to the importance of feedback and validation in a true demo culture.

My own conclusion is that hybrid approaches don’t work. At least, none of the ones I’ve personally experienced. Again: it feels counterintuitive because there are a lot of processes that appear to be complementary rather than in conflict, and merging them should be a 2+2=5 situation.

The reason hybrid approaches tend to not work is because they solve for different problems. Document-first approaches solve for the “why” - what work is worth taking on, why, and when. It answers business questions. Demo-first approaches solve for the “how”. They focus on experience, tangibility, and the specifics of execution.

Rather than creating a hybrid approach, you should selectively deploy one or the other method of building depending on the core constraints of the project.

Here are my rules of thumb/questions to ask:

Rule of Thumb 1: What’s your cost to build?

Cost to build can mean many things to many people, but I’m referring specifically to budget and the availability of people’s time and attention.

• If the “cost to build” is high (low budget, or your team has low bandwidth to take on projects), then you need to lean towards a document/validation-first culture of development to kill more projects than you approve.
• If the “cost to build” is low (large R&D budget, loads of engineers, lots of time), then you should lean towards a demo/ship-and-iterate style culture. If you don’t, people will be sitting around waiting for work.

Rule of Thumb 2: What’s your risk?

Every project carries some risk – whether financial, reputational, legal, delays, unforeseen problems, or otherwise.  

• If your risks are high (let’s say you work in a tightly regulated field, proposing a significant investment, working on a hard-to-reverse decision, or working on a flagship brand defining feature…) then you should lean towards a document/validation-first approach to de-risk your work. Pre-mortems are an especially useful tool here.
• If your risks are low, then you should lean towards a demo/ship-and-iterate approach. Getting things wrong in low-risk environments is a great way of learning!

… as a corollary to this, how much is unknown/unknowable, and how many assumptions are you making?

• If you are making a lot of assumptions or a lot about your project environment is unknowable (i.e., the work of invention), then you should focus on experimentation via a build-and-demo culture. Action produces information that no desk research or surveying can uncover because either the answer doesn’t exist or there is no right answer.
• If you are working with a well defined problem space and a lot of established research, save yourself the trouble: don’t reinvent the wheel, build on (and not against) solved problems. Preparation and pre-work will save you a lot of rework.

Rule of Thumb 3: What’s the size of effort?

Similar to “what’s your cost to build”, what's the size of your effort? Project estimation is loaded with all sorts of planning fallacies, but most of the time you can spitball an informed "this is a task versus this is a big effort" estimate.

• If the deliverable is effectively a single task (or maybe a set of simple requirements) that will take a few days or a week to build or requires a single person for delivery, then just go ahead and build it. You don’t need a lot of business scaffolding around it.
• If the deliverable is a large project, something that is expected to take multiple people, multiple weeks, and interacts with many different systems… then you should take the document-first approach to get everyone on the same page & figure out all of the dependencies you might not know about.

Rule of Thumb 4: What are you debating?

This gets to the heart of the differences between Apple’s and Amazon’s approaches: people are really bad at figuring out and making sense of abstract things, understanding probabilities, and debating relative merits.

• If the contention/alignment is around tangibles or experiences (such as interfaces, designs, or communications), then a demo approach is the way to go. Kocienda describes it as ‘trying to debate which imaginary puppy is cuter, a ridiculous debate until someone shows you the two puppies’. The better way is to put something in front of someone and to react to the specifics.
• If the contention/alignment is around strategy, business problems, goals, or similarly abstract concepts, then go with a document approach. In circumstances like this, jumping straight into demos or prototypes bypasses the important prerequisite questions… only for them to surface again more painfully later… Many doomed initiatives start this way and end with someone asking with exasperation, “why are we working on this again?”

Rule of Thumb 5: What are your team’s strengths and weaknesses?

Teams tend to be weighted towards one polarity of skillsets or strengths. Some people are naturally better at diving into building things while others are thorough researchers. It is hard to teach people to work against their natural strengths. So assign work to their strengths instead.

Instead of prescribing a one-size-fits-all process, focus instead on identifying the right problem-solution fits, set clear standards for outcomes, and maintain flexibility for how the team gets there.

What does this mean in practice?

Demos and documents are tools for solving different problems. Some projects may never need documentation, while others might skirt the demo. The best approach to planning and building things is to make sure the tools you use reflect the problems you're working on. That means processes can vary project by project, and you can change up your processes as you learn more about the problems at hand.

And that’s ok.

Continued Reading

If you're interested in exploring this topic more, I recommend the following:

Books:

“Working Backwards” by Colin Bryar and Bill Carr (about Amazon's processes)

"Creative Selection" by Ken Kocienda (about Apple's processes)

"Shape Up" by the team at Basecamp (About 37 Signals' processes)

"Creativity, Inc" by Ed Catmull (about Pixar's processes)

Articles

"Putting Amazon's PR/FAQs to Practice", Commoncog

"Creative Selection: A Summary", Commoncog

"Action Produces Information", Commoncog

"Apple Demos vs. Amazon Memos", StatPost by Trung Phan

"Project Management at Big Tech, and the Curious Absence of Scrum", The Pragmatic Engineer

"The Cynefin Framework For Decision Making"

"What Should Be On the Roadmap?", SharedPhysics

"Some Common Planning Fallacies", SharedPhysics

"Premortems", Asana