Why Pilot Projects Can Be Catastrophic

Many companies think they are staying nimble during product innovation by setting up pilot projects to validate concepts before they’re rolled out at scale. But pilots aren’t the answer, either, at least not on their own. Once something gets anointed as a “pilot,” it’s no longer an option—it’s the destination. There are typically no graceful ways to kill a pilot, and even course corrections are too hard to make. Systems such as software have all been done at the production level, with the assumption that the pilot will work and will need to be quickly rolled out at scale. Changes are seen as a sign of defeat, and digging into production code can be complicated. Besides, problems at the pilot stage often get hidden. A pilot is very public, and some senior people have a strong interest in success, so they may work behind the scenes and use their connections to make it successful. I once watched a client be all over a pilot in a single state, so thoroughly covering the pilot with senior-management attention that the client learned little before initiating a national roll-out. The executives knew what they were doing, but they couldn’t help themselves. They were so invested in the success of the pilot. The solution is to rephrase the issue. There needs to be less planning and more testing. The only way to accomplish that is to defer the pilot stage and stay in the prototyping phase much longer than most companies do. The difference between a prototype and a pilot is that there’s no possibility or expectation that a prototype will turn into the final version of the product or service. Prototypes are just tests to explore key questions, such as whether the technology will work, whether the product concept will meet customer needs or whether customers will prefer it over the competitive alternatives. The early prototypes should be all chewing gum and baling wire. They shouldn’t have hardened processes or the people required to go live. Yet they should provide real insight that informs further development. Each stage of prototyping should minimize costs and maximize flexibility. To borrow a term from computer programming, new products and services should be explored using “late binding”; they should take final form as late as possible, based on the most up-to-date learnings that can be generated. Pixar has made a religion of prototyping through what the company calls “story reels.” The company doesn’t just write a script; it creates storyboards that provide a sort of comic-book version of a prospective movie, then adds dialogue and music. The story reels cost almost nothing, compared with the fully animated versions of Pixar’s movies, yet provide a great sense of how a story will flow and allow for problems to be spotted. The story reels can also be changed easily. Here’s a fascinating video in which the creators of Toy Story describe their storyboarding process: https://www.youtube.com/watch?v=QOeaC8kcxH0#action=share Every regular review of progress on the prototypes should begin with a demo, much like what Pixar does with its storyboards. My old friend Gordon Bell, who designed the first minicomputer while at Digital Equipment, likes to say that “one demo is worth a thousand pages of a business plan,” and that notion applies to every stage of prototyping. It’s easy to get lost in talk of value propositions, competencies and market segments. A demo makes an idea tangible in a way that no business plan ever will. At Charles Schwab, in the lead-in to the company’s great, early successes with the Internet, executives talked about a hamster on a wheel. Schwab would test potential services by having people working behind the scenes answering questions, looking up information, and so on, running just as fast as their little (metaphorical) legs could go. Anything that didn’t work or didn’t resonate with customers was easily set aside. Only once Schwab had a sense of what customers truly wanted would it start building the capabilities into software. Prototypes and demos are part of what has made Apple products so successful. Steve Jobs always used prototypes of products to drive his thinking. For example, early in the process of figuring out the right screen size for the iPad, Jobs had Jonathan Ive make 20 models in slightly varying sizes. These were laid out on a table in Ive’s design studio, and the two men and their fellow designers would play with the models. “That’s how we nailed what the screen size was,” Ive told Walter Isaacson in his biography of Jobs. Admittedly, it helps when you have a genius like Jobs playing with the devices, but even he couldn’t envision everything. He needed many alternatives of something tangible. As Isaacson quoted him as saying, “You have to show me some stuff, and I’ll know it when I see it.” If Steve Jobs thought it was critical to prototype, shouldn’t you?