Test Your Ideas by Design

Tanarra Schneider, director of insights and planning at Critical Mass, talked to Andrew Benedict-Nelson of Insight Labs about how designers use prototyping to solve problems and what it might mean for other fields.

Andrew Benedict-Nelson: How would you define prototyping to someone who had never heard of the concept before?

Tanarra Schneider: The way I look at prototypes, regardless of whether it’s a digital or physical prototype, is essentially building a rough but functioning sample of whatever the finished product is going to be. From a digital perspective, it might be parts of a website that are clickable and would give you some semblance of the real experience. But it could also be something like a mock clinic, where you actually find a space and walk people through what it might be like to be there. Or if you were prototyping something like the next iPhone, the most relevant features might be the weight, the size, the feel. The whole point is to help people experience the key elements.

Andrew Benedict-Nelson: Okay. I’m guessing there are plenty of folks in the world who are developing products and say, “My God! We don’t have time for that! We have to get to market ahead of our competition!” How would you justify prototyping to someone who took that position?

Tanarra Schneider: Actually, it depends. That’s not a point that isn’t valid. There is this notion of speed to market, of getting to market as quickly as possible to make as much money and possible and just seeing what happens. But the thing is, the first version of whatever you sell will also be your first live prototype. Things will be broken. Things will not function. And you’ll get feedback about those things.

So if you’re diligent, you could quickly incorporate that feedback and turn around the next version. That may work for a digital product. For a business or a physical device, the cost of getting it wrong the first time may be substantial, especially in a business environment in which people demand continual improvement. So even though you think you’re going to corner the market and make a lot of money, you actually end up wasting money because you didn’t take the time to do something like a limited release to see if the thing actually works.

There’s something else that prototyping allows you to do, though. It allows you to pivot. If you do a prototype and find out that what you were building is not exactly what the market needed, you have an opportunity to pivot before you’ve invested all of your time or energy into a single concept. This is a consistent story. It’s the story of a number of companies that Eric Ries started. It’s the story of how Flickr got started. The pivot sometimes comes at great cost, but it may also come at little or no cost, and could save you a lot of time and money when you do a wider release.

Andrew Benedict-Nelson: I’m reminded of pharmaceutical research. Several popular drugs were originally developed to do things that are quite different from what they’re marketed as doing now. But because they had to go through such rigorous testing, they discovered that they had these other useful effects. You could view the market in a similar way. It can reveal something significant about a product that you didn’t see before, if you’re in the position to see it.

How do you prototype something more conceptual, like an organization or an experience?

Tanarra Schneider: I’d say you still have to prototype it, especially if it’s an experience. Let’s go back to the clinic I mentioned. Kaiser Permanente did prototyping with clinics in Oakland. What they were trying to capture was how people perceived everything from triage to actually receiving care. They used that exercise to improve upon their existing clinical experience.

If you are setting up an organization like a consulting firm, you should think of some key things that organization will do. Why not walk through a mock project together? Why not find someone who is willing to play a client, or who could even be a client at some point? You can get some sense of their reaction or figure out where they may have missed a step. You can see where you may need to do more work to explain your methodology.

So I think there is almost always a chance to prototype a new organization. I can’t imagine that someone would start a charter school without walking through the way classes are run or how the administration will work or how children will come and go from the facility. There are so many options to live prototype experiences to make sure that the people you are serving will get what they need.

When you think about it, it’s actually more important than prototyping a device or a digital tool. People have more tolerance for that. They may come back even if their first experience isn’t perfect. But when you think about an organization or an experience, do people really have the tolerance to come back again if their experience isn’t right the first time?

Andrew Benedict-Nelson: But we know, no matter how sensible everything you’re saying may seem, that people actually do open schools and offices and organizations without any type of prototyping at all. Why is that?

Tanarra Schneider: I think we tend to come at these things as experts. If I am setting up a school, I’m a teacher, I’m an administrator — I know how things are “supposed” to be run. Kids don’t know anything. Students don’t know anything. To me, it’s that assumption of ego, of expertise, of “we always know best.” We don’t want to take our egos out of it, even if we might discover things that could help us improve on an experience. It’s a hit to your ego when someone who is not an expert in your field comes in and says, “Wait, no, this is all backwards.”

Andrew Benedict-Nelson: Could you tell me a little bit more about where prototyping fits into the design process as a whole? What are some things that would happen before and some things that are likely to happen afterward?

Tanarra Schneider: Well, there can be multiple moments of prototyping — it doesn’t have to be just one shot. But typically in a design process, you would do a lot of investigation and exploration up front. If you’re a good designer, you’ll figure out the problem you’re trying to solve, then you’ll talk to some of the people who are impacted by the problem — what are their needs? How are they affected by this? Where are existing solutions falling short? You can also ask similar questions of potential users of the solution or product.

That’s all research, but typically you would have an actual design of some sort before you do prototyping specifically. You would at least have some models or sketches. For example, in a recent project, we actually did two phases of prototyping. First we showed users some sketches of what we wanted to do mixed in with some of the things that were already out there. Then our user-experience designers and visual designers used that feedback to create a second prototype that was actually more clickable and true to form. It was still not fully functional — it was representative of a more complex tool that we’re building.

So you do these iterative phases of discovery, learning, designing, and building. You’re constantly putting things in front of people and getting their feedback. That also helps you think of the first version of the real product as the next prototype. But we’ve already weeded out the risk from some bad ideas and capitalized on some good ideas. If you’re good at that, you see it all as part of one iterative process.

Andrew Benedict-Nelson: I’m trying to figure out where that perspective comes from. It seems like in addition to the desire to weed out risk, you must also need that sort of Underwriters Laboratories spirit where you test things until they break. Where does that come from? Not everyone has that personality type.

Tanarra Schneider: I hope the people without that personality type are not hoping to become user-experience designers. But you can still do it without that personality type. You just have to genuinely care about the people who are going to use what you’re designing.

Whether you’re a designer or a teacher or an attorney or a doctor, I would hope that if you are going to build something from the ground up, that you care enough to understand how it impacts the people you say you’re trying to serve. You have to challenge yourself to not be afraid, to get your hands dirty, to understand the guts of what you’re building. Otherwise, when real people are using it, and it breaks, you can’t fix it. Even if you just want to optimize it or tweak it, it may be too late to fix it.

Andrew Benedict-Nelson: I think it’s interesting that empathy with users provides you with this sort of shortcut to design thinking.

Tanarra Schneider: Almost everything we do starts from the space of empathy. It’s empathy with users that makes it seem natural to do all of this tinkering and taking things apart. It’s weird, because we can be inquisitive to the point where it makes people uncomfortable. But it doesn’t come from any sort of inherent interest in insurance or printers or hospitals or whatever. It comes from a desire to serve the end users of whatever system we happen to be working on.

Andrew Benedict-Nelson: Can you tell me about a situation where were it not for prototyping the design would clearly have failed?

Tanarra Schneider: Oh God. There are so many. I remember a story about the testing phase of an insulin delivery device. One of the biggest issues was that the device made a horrible clicking sound when you pressed the button for delivery. It was because they were testing a new kind of pneumatic mechanism. But it sounded horrible. As soon as they heard it, the patients would back away.

So you’re talking about someone potentially stabbing themselves or wasting medication. It’s a health risk if they’re not getting the dosage they’re supposed to get. That was discovered in prototyping and testing in the lab. And they were able to go back and make some minor changes that made the device usable. That prototype probably saved them millions of discarded units and also saved lives.

Andrew Benedict-Nelson: It seems important to observe here, though, that there’s also a lot you can learn from prototyping even if you don’t encounter some fatal flaw in the product.

Tanarra Schneider: Yes, there is always something to learn. Even if what you learn is that 98 percent of the design is right on track, you may discover that there is another two percent that a significant group of users focus on. It could be something as small as the image for an icon. It’s worth doing if it finds anything that can help your product or experience reach its full potential. Even if you can’t get whatever you learn into the current release, you may get ideas for the next release.

Andrew Benedict-Nelson: So you can go in with a critical eye but don’t necessarily have to pull the emergency brake.

Tanarra Schneider: Right, and I think the idea of that emergency brake is a big reason why people fear this kind of testing. The product may be in production or the experience may be out there, and they ask themselves, “What’s the point of testing now?” But there’s always a point to testing if you think there is something you could learn. You could get in front of issues or think of the next thing that could delight your users.

Andrew Benedict-Nelson: And it’s not as if you’re totally helpless. Let’s say you were doing that testing on the insulin device and a bunch of them had just shipped to a refugee camp. Well, you could still issue instructions to the doctors there to warn people that the device makes an unpleasant sound. You would want to know about that before you actually lose lives.

So when you look out at the non-designers of the world, how are we doing at prototyping?

Tanarra Schneider: It’s funny — you can hardly ever tell if a designer was involved with something or not these days. When you’re talking about a product, you can usually assume there is some sort of product engineer or designer who would have a stake in this kind of testing. But when you leave the space of design, I think the problem is that for a lot of people, admitting that you don’t know what you’re doing is the same as saying that you’re failing. There’s an assumption that if you have to test it out, you are afraid of failing.

Look at something like “America’s Test Kitchen.” A lot of people don’t understand why they would try out 100 different kinds of tiramisu. They think, “Just make the tiramisu.” They don’t get the passion that drives someone to understand everything that goes into it.

I’m also surprised by how little testing we do on things that really matter. I have a passion right now for understanding health care. I don’t see a lot of people prototyping anything there. I know some doctors in New York who are experimenting with the locations where care is delivered. They are using Google Calendar and coming to people’s homes. That began as an experiment where they didn’t know if it would work at all, then became a version of a prototype. These doctors have admitted that they’re trying to think more like designers, trying things out and then trying them again.

Andrew Benedict-Nelson: I think the critical component of what you’re saying is that it’s all about mental attitude. I can understand the situation of a physician who says, “You know, I can’t afford to run experiments on my patients.” But there are situations where life provides you with some natural experiments, and you can treat those at prototypes even if you’re not a designer.

It’s tough because there’s less control. Think about your test kitchen example. Imagine you tried the sorts of things they do in a test kitchen in a regular kitchen. You walk in and someone’s counters are covered in twelve different types of brownies. A lot of people would just assume that they had screwed up eleven times.

Tanarra Schneider: Exactly. And that’s too bad. I mean, I like to cook, and I spent the better part of a year trying to perfect a pancake recipe. I knew it wasn’t quite right. I knew it could be better. But the number of people who bring that mindset to their daily lives is pretty small. Probably because it can drive you crazy.

Andrew Benedict-Nelson: But this also reflects a real challenge in the design process. How do you know when you’ve done enough testing, when you’re ready to just ship it?

Tanarra Schneider: It’s all about finding those couple of criteria that are really, really important. Otherwise you’ll test every possible permutation. You have to test for what matters. You can do that by thinking about the goals of the product or experience, then asking what’s really critical to making it work. That also helps  you make sense of the things that happen while you’re testing. Not every comment that someone makes will be important. At some point the color of the navigation doesn’t matter as much as whether the person can use the navigation. So what’s the core? What’s the heart? Testing that is what has value.

You also need to think about who will be using whatever you’re putting together. Some people think that you need to satisfy every possible group of people in a test. But I’d recommend focusing on the 80 percent of people who you think will be using the product most of the time.

Andrew Benedict-Nelson: It’s interesting how that idea of setting priorities is implicit in testing. That could also help you make sense of what you’re doing.

So let’s say that you were about to talk to a group of non-designers who are completely persuaded by everything you’ve said and are about to prototype something for the first time. What would your advice to them be?

Tanarra Schneider: A common problem is what you were just talking about: prioritization. Everyone has their favorite feature, their baby. So do your due diligence up front. Talk together about what the priorities of the product are. Then distill them into clear, measurable, testable outcomes. Start as small as you can, trying to test for as few of those as possible. You can always do simple tests again. What you can’t afford to do is generate a lot of data that you can’t use, since someone always has to take the time to go through all that data and distill it into something meaningful. If it takes you six months to analyze all the data from your prototype, will it really help you that much?

So don’t try to conceive of a NASA-inspired test the first time out. Do something with as little hassle as possible.

Andrew Benedict-Nelson: Let’s say that you’re talking to a person working with an organization or culture that is not likely to adopt prototyping across the board anytime soon. What would be your advice if that person wants to initiate some prototyping on their own?

Tanarra Schneider: Look for a Skunkworks project, something you could do on your own or with one or two close supporters. But look for opportunities that will be meaningful to more than that small group of people. You’ll not only have to adopt that mindset of taking things apart, but you’ll probably have to do it on your own time, on the weekend. But if you really believe in what you’re doing, it tends to sway people pretty well. If you can quickly share a few things you learned that can improve the organization, and you don’t expect anything back, you can change people’s minds.

Andrew Benedict-Nelson: It’s interesting – we’re used to the standards of scientific truth, where you need to control every variable. But there are opportunities to do prototyping that don’t look that way.

Tanarra Schneider: Right. People need to stop thinking of the prototype as this monolithic thing or this showstopper moment where they prove that everyone else was wrong. They should instead think of prototyping as a series of small opportunities to learn that are coupled with our ongoing duty to the people we serve. That makes experimentation a lot more valuable and a lot more meaningful.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. It is attributed to Insight Labs, and the original version can be found here.