User Science is Product Management

Brent Tworetzky, Executive Vice President of Product at XO Group, shares his philosophy of user science, and explains how to assess customer perceptions and behavior when developing new products and features.

Here are the highlights:

• What experiences lead Brent to his product management philosophies? (2:26)
• What is User Science and how does Brent apply it at his organization? (3:45)
• How does Brent uncover actual user behaviors and how does it lead to more meaningful insights? (7:30)
• What are the challenges associated with user science? How can product teams get started? (9:12)

And here's the transcript:

Mike Fishbein: Hey, I'm your host Mike Fishbein. This is Product Management, a podcast produced by Alpha. Welcome back. On this episode, I'll be speaking with Brent Tworetzky, the executive vice president of product at XO Group. We'll discuss his philosophy of user science and explain how product teams can assess and distinguished between user intent and user behavior.

Brent Tworetzky: Hi, everyone. My name is Brent Tworetzky. I'm the EVP of Product at XO group. We're known for the Knot and the Bump, leading site snaps for wedding planning, pregnancy, and parenting. Today, we're going to be talking about user science, which is the study of users. It's a combination of both user research and user analytics.

User science is product management.

Mike Fishbein: Brent has a really interesting framework that I'm excited to dig into. Before that though, let's learn more about Brent's background and how he got into product management in the first place.

Brent Tworetzky: I started my career as an engineer, as a programmer in high school and college. I love the power of tech and the power of tech to change the world in multiple different ways. Immediacy, personalization, scale, though I never felt that excited about the actual work I did when I was a programmer. This was back in the late '90s where programmers weren't the hacker heroes that they can be today.

I shifted over to the strategy side. I did some management consulting, some account management, and some venture capital to try to get more into the why of what we can do with technology and the strategy behind it and help the side and figure out what to do in those roles. I loved what I did on a day-to-day basis, but I felt removed from the action.

Product management to me was a way of combining what I loved about getting my hands dirty and technology with being involved with the strategy of what to do. It was sort of this magical a-ha moment of putting all the things I loved together into one role.

Mike Fishbein: Brent has a great blend of experiences that are critical for product management. How has his perspective of the discipline evolved through it all? How did he devise the philosophy behind user science?

Brent Tworetzky: I've been very lucky to work under some great chief part officers. My first role in product management was as a PM at Mint.com, where I learned from ahead of products who had been at eBay. eBay is a great product management philosophy that's very business case driven, very quantitative, and results-oriented and I learned a ton from him.

My second stint as a product manager was at a ed-tech Company called Chegg where I learned under the chief product officer from Netflix. Netflix has this awesome user research DNA where they train people to use leading edge tools to understand their users, so I basically combined those strands of the Netflix product DNA and the eBay products DNA to help guide a lot of my product thinking.

Lastly, when I ran product at Udacity, I worked with a founder and a head of engineering who were from Google and I combined some of the Google ways of thinking of collaboration and user understanding and quick iteration, grafted that DNA onto my approach to this, into this emerging field of user science.

Mike Fishbein: Brent has grown and learned from product leaders from eBay and Netflix. How does he define and apply user science at his organization?

Brent Tworetzky: Product management in its modern form is a combination of strategic thinking, execution and user science, user understanding, so that's helping us figure out which direction we should go in, helping work with the team to make the product come alive, and then understanding the users to help shape and direct future directions where the product should go, either should we iterate it and improve it where it is today, or should we go into a completely different direction? User science is this magical field of deeply understanding the user so we can build the right things for them.

It's a combination of understanding the user's intent. What needs do they have? What problems should be solved for them, as well as understanding user behavior? What do they actually do when you put a product in front of them?

And there are a couple of ways of thinking about this. You can in both the intent side and the action and behavioral side, we can both do a deep dive. I was sitting in front of them, whether we're interviewing them for understanding their intent or whether they're observing them, through usability testing to understand their action or we can do some kind of big ranging kind of study. Under intent, we can send a survey to users under action. We can look at their aggregate analytics or we can perform A/B testing, so it's almost like looking at it as a two by two and figuring out which tools should we use in which situation.

We have a couple of other tools in our toolkit that span across these different areas, such as collecting customer research and customer support data such as beta tests and as a diary and longitudinal studies.

Mike Fishbein: User science encompasses intent and behavior to identify what value proposition resonates and what features gain adoption across the user base. There are different tools and practices when approaching each.

Brent Tworetzky: We always want to target the user science tool to the problem at hand. So for example, in a new product, we always want to start by understanding the users and their needs. By understanding their intent, we might start with some interviews, observations, trying to figure out what problems is the user trying to solve in their current day. What tools do they use to solve that problem or how do they compensate for that issue?

We might then take some insights from one on ones in small groups and expand on them with a survey to confirm that what we're interested in and what we found is actually the right answer.

You can go back and forth between small groups and one on ones as well as surveys to truly nail down what the important problem to solve is. Once you then actually have the problem to solve and you have a good direction where to go, you then might move to the user behavioral side of user science.

You might build a prototype and put it in front of them to get some usability testing. You might put a couple of different versions in front of them and use A/B or bucket testing, or you might look at aggregate analytics to see how they're behaving.

When you think about these two sections of user intent versus user action, the user intent set of tools, allows you to look at everything that's out there, any option is possible to solve a user's need.

When you think about the user action side of the tool kit, you're really looking at the iteration side. You have something out there and now you're looking to figure out does it work or not? Should you improve and continue to iterate on it? Or should you throw it away and start again?

So depending on if you think you've found a real problem and if the solution that you have solves that problem, you might iterate between any of these tools, go back and forth between small scale and large scale, or between intent and action.

Mike Fishbein: Brent mentioned a really interesting technique his teams use to uncover actual user behavior. I asked him to explain it further and illustrate how it leads his organization to meaningful insights.

Brent Tworetzky: One of my favorite user science tools is the diary study. We collect a small group of users who agree to share their thoughts and challenges with us on a regular basis over an extended period of time.

For example, if you're solving the problem of wedding planning and the average wedding planning journey takes you a year, we might check in with a couple or a member of the couple every week or every two weeks over the course of six months, 12 months, to understand what are their needs and challenges that week.

With the diary study, you start with a set of hypotheses such as, I want to understand what tools they use, how much they want to spend, what problems do they have, an idea that I have to solve their problems. And then you ask the questions during the course of that study to observe them, understand them, and identify do they really have the problem that you're searching for.

I like diary studies that are directed at user understanding to truly understand what their needs are. Sometimes users struggle to articulate a need or a problem at any particular point in time, at the beginning or middle or end of the journey, and it really comes through watching them through the diary study over the course of weeks and months to really understand what their needs are.

If a user says, "Man, I had this problem this week," and they repeat that again and again and again, and multiple users do that, then you've really found a problem to solve.

Mike Fishbein: In theory, user science sounds really great, but how easy is it to implement and operate in practice? What challenges might a product manager or team run into?

Brent Tworetzky: User science is a field of product management that needs to be learned. It's not natural, such as good communication or collaboration. At XO Group, we train all new team members on these skills by explaining and walking through the philosophy of how we might use these tools in practice.

We also then ask them to put these tools in their hands and do practice tests themselves. For example, everybody who joins gets tutorials from either our user research team or our product analytics team on how to use Optimizely for A/B testing or UserTesting.com for usability testing, and then we'll ask our team members to practice tests and we'll give them feedback on those tests. This way, our team members get familiar with the tools and build muscle memory about what those tools are.

We also are constantly bringing in guest speakers into our company to teach our team about the capabilities of new tools and how they can use these tools to make magical things happen for our users.

There are some user science tools that would be very difficult for an individual product manager or product designer to do by themselves, such as a beta test or a diary study, and we have some dedicated people on our team to run these time-exertive and specialized tests.

It's also helpful to note that we can't just tell people about user science and expect that they get it overnight. It's truly a field to be learned. I've found that there's four learning curves for user science. The first one is just awareness of what all the tools are. Not everybody is familiar with surveys, one-on-ones, beta tests, usability, A/B tests, etc.

Once people actually understand all of these tools, then there's a second learning curve about actually being able to do the tests with quality. Often when people learn about new tests and they practice them for the first time, they'll make rookie mistakes. It's easy to accidentally bias a test, an A/B test, or a survey or an interview accidentally, and so somebody needs to learn how to actually execute a test without bias.

The third learning curve is around the appropriateness of tests. Sometimes you'll find that team members will use the test that they're most comfortable with, but not the right test for the situation. And lastly, product managers need to learn how to use tests in a valuable way. It's not just enough to know what the tests are, use them correctly, use them in the right situation, but it's also important to be able to gain value out of the test that you do.

For example, I've often seen in young product managers that they'll always do an A/B test for any new idea against the original as a sort of protection or safety net, and that's their go-to.

What I encourage people to do is to use tests and to be more provocative, to stretch the boundaries of what they're testing so that they can learn more. After all, the cost of the crazy test is just a small impact on your users for a great amount of learning.

Mike Fishbein: Brent illustrates the trajectory and maturity phase that product teams must go through when adopting user science. Ultimately, what will be the impact on their organizations and what advice can Brent share for those interested in getting started?

Brent Tworetzky: Some of the most fun about user science is just practice, practice, practice, spending time with users, and discovering new sides of them. The field is also advancing quickly. It's often misunderstood by people outside your organization, so you might need to be an advocate for the field to get other people to be involved. Generally, when others in your organization know that you're learning more about your users, they're excited to participate as well.

My first piece of advice is to hire and mentor fantastic people. You're constrained or amazed or limited by the people around you, and so making the people great and hiring great people, and helping them get better sets your environment to succeed.

Provide a clear mission and vision and strategy to align your team so that everybody's moving in the same direction. Sometimes at early-stage companies, everybody knows where you're going, but the bigger you get, the more important it is to have clarity so that everyone knows where you're going and works as a team.

Make it clear to your team what you expect from them, what success looks like from them in their roles now and in the future. I subscribe to Daniel Pink's philosophy on drive and it's a great book if you haven't read it. He basically says that people derive meaning in their work if they have room for autonomy, opportunities for mastery and a rich sense of purpose. So, I use that acronym AMP and think about that constantly to create a great team environment.

Create a great work environment and process that focuses on outcomes. It's easy to focus on output and celebrate a release. It's more important to get people to focus on outcomes and actually making things that matter for your users and for your business.

And I guess the last piece of advice I'd give on leadership and management is to communicate, communicate and communicate. There's no such thing as over-communicating, especially the bigger you get.

Mike Fishbein: User science is a practical and valuable framework for product teams to align their efforts with customer perceptions and behavior, but it takes a lot of work, communication, and expectation setting before your team gets to full speed.

Next up is the benchmark. Let's see how Brent reflects on the next series of questions we asked all interviewees to ask themselves.

Brent Tworetzky: How do I eat my own dog food in user science?  I treat my team like a product and apply some user science to it. Anonymous user feedback, performance analytics for myself, testing new things and I actually have my team members score me so we keep the things that people like such as our regular product school, we get rid of things that people don't like, such as fewer meetings, add new things that people will suggest. We tried something new recently called peer PMing, and overall my org is changing in a relatively agile way. I think it's improving. People seem to be happier, getting more meaning and success from their work.

How do I get out of the office? Oh wow, I love users and that's part of what drives a product manager. I love to validate emerging research with my users if I can, so some of my favorite things to do are visiting wedding vendors and talking to them about their challenges while throwing new ideas out at them.

We talk to and visit users all over the country, both couples to be, brides and grooms to be, florists and wedding venues and parents to be and just traveling the country and meeting users in different areas ... It's just always one of my favorite parts of my job.

What am I reading right now? I recently finished the Google Ventures Design Book on design sprints called Sprint. We had it as a book club with a lot of my other product leaders and now we're talking about how to test making it a regular part of our process.

This is an actual nightmare that I have sometimes and it's mostly about missing the boat on what matters to our users, which is both products that we might make that don't work or not changing as our products change. Those are the things that really keep me up at night, that we're not solving their real problems.