How to define and drive product quality

What I learned about product quality while working on Airbnb Experiences

May 26, 2021

Later today I have the honor of “speaking” (ie. having a pre-recorded talk played while I hang out and answer questions in chat) for a group of product builders at the Women in Product conference. The topic? How to define and drive product quality.

(If you subscribed to this newsletter after “attending” my talk, thanks for coming and bigger thanks for letting me promote myself. Always awkward!)

So back to quality.

QUALITYYYYYYY.

Quality is not a word I often hear during the product development process. Yet, as a Builder, it’s something that I’m working toward creating every day. Product decisions compound in wild ways, and it’s interesting Builders don’t talk about this more because it’s an assumed given. Who sets out to ship a low quality product? Not me. Hopefully not you!

In the imaginary world in my head where I have all the frameworks and tools to do my job well every day (Builder Nirvana!), I’d have some sort of quality playbook that I could pull out and thumb through at key moments of my process. Unfortunately I don’t have this and I’m guessing you don’t either. The art of balancing shipping “quality” with shipping early often falls squarely on your shoulders, dear Builder. So let’s talk about what quality is and unpack some different ways in which you can intentionally create quality with your team.

What is product quality?

One of the problems with bringing teams together around a shared idea of quality is that it can often mean something different to each of the people on the team. I unscientifically polled (ie. texted) some product managers and engineers I’ve worked with over the years and when I asked them to define quality they ALL said something different: quality assurance, the desired customer experience, test-driven development, quality = trust, and I quote “bug hunts baby.”

Cool!

I also looked up an industry definition for software product quality and found nothing super compelling that wasn’t focused on making sure you’re not shipping bad code a la CI/CD, test-driven development, or creating physical products (like building cars via six sigma). Most of the industry definitions revolve around quality assurance which comes late in the development process after the thing is mostly built. The majority of folks I spoke with brought up the end of the development cycle where you’re making sure that what you’re shipping isn’t broken.

There’s the rub! Instead of checking for bugs at the end, you should design a process that bakes in quality from the start.

So let’s look at how you can think about this throughout the product development lifecycle when you’re building the damn thing.

Product quality defined

Boiled down, product quality is an output. It’s a blend of tangible and intangible characteristics that, when combined in the right way, hit a sweet spot on your user’s value stick. It’s hard to define because that definition of quality is variable to different users, different products, and different tastes.

See, you start to understand why product quality is so squishy. Start with the customer and work backwards to make sure what you’re building is right for the people using it. This is simple in theory but not always easy to align a team around and to realize.

Define and drive your version of product quality

But let’s try! Start by defining your quality north star. This should be an outcome that you think clearly communicates quality. This outcome should help your team create the right experience for your user. Here are some tangibles and intangibles to get your juices flowing.

1. Align on your quality north star

Real-life example time: Airbnb Experiences north star
Let’s look at a real, in the wild example of how to select a quality north star. A few years ago, I joined Airbnb as a product manager on Airbnb Experiences right after it launched. It took us a while to find product market fit, but once we had it we began to scale up quickly and set aggressive growth goals. Buuuut we quickly realized that by scaling too fast we had brought on hosts and Experiences that didn’t quite meet our standards. The problem was we hadn’t really articulated those standards to the teams building the product and hadn’t really aligned on the right quality north star.

Because it was a brand new marketplace, building trust through a great experience with new customers was critical, and we knew if we didn’t course correct urgently, the entire platform was at risk. Considering this, we agreed on a clear high-level quality measure: x% five star reviews. And we deliberately set it high. This was a great north star, because it helped the team make decisions in the interest of quality. It was also a great check to our other growth goals. Not all growth is good growth.

Setting your own north star

So based on what you think is a clear quality outcome, set a goal around it. A good output goal:

Is measurable, but more than just a metric
Is one of the highest priority outcomes you’re trying to achieve (ideally there are less than 3)
Can create healthy tension with another outcome that isn’t quality related

Here’s a look at how the inputs ladder up to your quality north star

2. Select inputs that drive that output

So once you’ve got a good quality north star locked, figure out all the inputs that drive that output. Ask your team, what are the ways in which your product needs to work to deliver on that outcome? Define the different drivers and set those as inputs. A good input:

Is measurable, but doesn’t have to be a metric
Is proven (or can be proven) to drive the north star outcome
Is highest leverage, or one of a handful of inputs that will have the biggest impact

Real-life example time again: Airbnb Experiences Inputs
So after we had decided that % five-star reviews were what we cared most about, we set out to unpack how to actually make those high-quality experiences happen. We wrote down a long list of factors that we thought would have an effect, but we decided to focus on just a couple that we knew we could control and that we thought would make the biggest impact.

Through research and taking many Experiences ourselves (dream job, really), we determined that the caliber of the host taking you on the Experience was everything and that the way we presented/sold the Experience to you on Airbnb mattered a great deal as well. Both of these things impacted trust and were things we could control, so we mapped it out and it looked a bit like this:

Here’s a look at a handful of inputs we impacted through the product experience

Bringing it all together

So after defining inputs, we socialized with our teams and got to work to build a product experience that focused on each individual input. While I love to simplify years worth of work into two paragraphs 😆, the hard truth was that it took a meandering path to get where we wanted to be. It was well over a year's worth of work, many team deliberations on what we cared about, experiments ran to understand true drivers of quality and much much more. It was a process, one that will profoundly impact the quality of Experiences on Airbnb’s platform for years to come.

So, based on what you learned here today, I’d very much love to hear about the different ways you and your teams are defining and driving product quality. Furthermore, here’s a Miro template to duplicate that takes the above framework and makes it a bit more actionable. If you use it, please share back how you’re using it in the comments below!

Builders