Do Your Product Manager Interviews Miss These Critical Analytical Skills?


You’ve always heard that you should hire the smartest product managers you can.

Some even say you should only hire people that are smarter than you.

But what does all this really mean?



Should you be trying to give candidates a verbal IQ test? Or should you be testing more practical problem solving?

What are the right questions to ask?

It is true, of course … you should hire the smartest product managers you can. And doing that well starts with a purpose-built interview.

 

The Analytical & Cognitive Skills Interview

The purpose of this interview is to evaluate problem solving and critical thinking skills. A product manager needs to be extremely smart, and this interview gives you the framework to effectively evaluate raw intelligence, creativity and analytical abilities.

Your goal should be to evaluate the candidate’s intellectual horsepower not in the context of esoteric puzzles and brainteasers, but in the context of common and relevant business problems.

This interview should take 45-60 minutes to complete. The key sections of this interview are:

  • Market Sizing & Problem Solving: 15-20 minutes
  • Business Model Exploration: 15-20 minutes
  • Metrics / Data Analysis & Problem Solving: 15-20 minutes

 

Market Sizing & Problem Solving – High Level Thinking

Product managers need to be able to quickly make assumptions and generate back-of-the-envelope estimates. Hypothetical market sizing is one of the most common ways of assessing this skill.

Good answers require a candidate to communicate key assumptions and keep everything straight while quickly doing the math in his or her head.

Interview questions you could ask include (you should only ask one):

What is the total annual revenue of all gas stations in the United States?

What is the average weekly revenue of a typical movie theater in San Francisco?

How many people travel from San Francisco to New York every day by commercial airline?

These type of market sizing case interviews are used heavily by top management consulting firms like McKinsey, Bain and BCG.

Here is a 9 minute sample interview video from Bain’s London office, where you can see how an interviewer leads a candidate through a market sizing case and how she analyzes his responses at the conclusion of the interview:

 

Business Model Exploration – Case

Winning in business is just as much about great business models as it is about features, technologies and value propositions. Unfortunately, not all product manager interviews focus as much on the former as they do on the latter.

There is a lot more to a great product than just it’s value propositions and features. Alex Osterwalder, author of the Business Model Generation and creator of the Business Model Canvas, talks about four levels of strategic mastery:

  • Level 0 – The Oblivious: Focuses on the product and value propositions alone instead of the value proposition AND the business model.
  • Level 1 – The Beginners: Uses the business model canvas as a checklist.
  • Level 2 – The Masters: Outcompetes others with a superior business model where each business model element reinforces each other. Examples are Nintendo Wii, Nespresso and Dell.
  • Level 3 – The Invincible: Continuously disrupts themselves while their existing business models are still successful.

To be most successful, you do not want your product managers to be playing at Level 0 or Level 1. They need to be comfortable iterating on the whole product and not have a toolbox limited only to features, technologies and value propositions.

In this section, you will evaluate the frameworks and processes a candidate uses to analyze a potential business. What you are looking for is familiarity with a wide variety of business models, a good framework for analysis, and creativity and divergent thinking in coming up with solutions.

A good framework for early stage startups is Ash Maurya’s Lean Canvas. A good framework for more mature businesses is the Business Model Canvas. Either of these could be complemented with the Eliminate-Reduce-Raise-Create grid of Blue Ocean Strategy to further develop interesting and disruptive ideas.

For a junior role, use a white board to analyze and iterate on business model prototypes together. Consider the analysis of either existing businesses or potential new businesses, depending on the stage of your company and what the product manager’s role will be. Have the candidate walk you through his or her process for analyzing issues and making decisions.

As an example, here is a video of Alex Osterwalder at Stanford using the Business Model Canvas to do a 5 minute analysis of the Nestle Nespresso business model:

For a more senior role, the candidate should be able to synthesize a framework and lead a discussion without help. Provide a more open ended scenario like:

Based on what you know, how would you identify and validate the best business model for us to pursue for one of our products?

Another good option for a startup specific interview question is:

Imagine you just received a $5 million Series A investment from Andreessen Horowitz. Describe your process to validate the proposed business model and to get the company to product/market fit.

This scenario not only allows you to test a candidate’s business model exploration and prototyping skills, but also tests for big picture thinking and comfort level with uncertainty.

As the candidate walks you through a business model design and prototyping process, the key areas you want to get clarification on are:

  • Revenue model
  • Gross margin model
  • Operating model
  • Working capital model
  • Investment model

These five elements were highlighted in Getting to Plan B: Breaking Through to a Better Business Model by John Mullins of London Business School and Randy Komisar of Kleiner Perkins Caufield & Byers.

For each of these elements, ask a candidate:

Talk to me about the [revenue, gross margin, operating, working capital, or investment] model.

Why did you choose this model?

Are there alternatives that might provide a better competitive advantage?

Here is a 14 minute video of John Mullins giving examples of companies with innovative approaches to each of the five elements:

 

 

Metrics / Data Analysis & Problem Solving – Detail Thinking

Smart product managers use data effectively to make better decisions. Ask about data analysis using an open ended question like:

Tell me about how you use data to make decisions.

What you’re hoping to hear is the candidate talk about dashboards and how to use actionable metrics instead of vanity metrics in the analysis and measurement of success. If the candidate does not get here alone, prompt him or her more directly:

Can you take one of our products and talk about how you would build a dashboard for it?

What metrics would you track? Why?

If you’re an early stage startup and haven’t released a product yet, pick a product the candidate is familiar with.

You want to hear a candidate talk about lifetime value, per-customer metrics, events, funnel analysis and cohort metrics.

Ask the candidate to step you through a funnel analysis on the whiteboard:

Can you walk me through how you might do a funnel analysis for one of our products?

Rapidly and clearly doing this exercise will tell you a lot about the candidate’s ability to think on-the-fly and how smart he or she is about data analysis.

Your ideal is a candidate who can talk about actionable metrics in the same way Neil Patel, Co-Founder of Kissmetrics, does in this 10 minute video:

 
 

 
P.S. A lot of people have asked for book recommendations to further prepare for a product manager interview. Here are some great ones:

Looking for a great product management job? Try Hired. It’s a free service where companies apply to you, not the other way around. Use my referral link and get a priority review.
 
Are you hiring? 200+ top candidates are waiting for you on Hired.


 
 

Andy Jagoe About Andy Jagoe

Andy is the founder of Venturegrit and has 20 years of experience founding, funding, and leading venture funded startups. More about Andy's background. You can also find Andy on LinkedIn and AngelList.

Get free and fresh Venturegrit knowledge as soon as it's published:


We will never share your email address with anyone.

Comments

  1. Ziv Rotner says:

    Andy, thanks for the excellent article !. Lot’s of materials to read over the weekend.
    In case you’ve prep’ing an ebook with more examples, I’ll love to have a copy.

    Ziv

  2. Great article. Very helpful from both ends of the hiring equation. One small item to point out:

    Why do product managers need to “quickly do math in [their] head[s]”? Mathematical and problem solving ability are not necessarily related to fast computational skills. Knowing how to set the problem up and solve for it is much more important than being able to do the arithmetic quickly without writing it down/using a calculator.

    • Andy Jagoe Andy Jagoe says:

      Hi David. Thanks for the comment. I agree that knowing how to set up the problem and solve for it is more important than being able to do math in [their] head[s]. However, a product manager needs to be able to move and respond very quickly and doesn’t always have the luxury (or wastes time if he/she always needs) to step away before responding. A product manager that can quickly do math in his/her head can react immediately in meetings and conversations when suggestions that might not make sense are made by senior executives or others outside the immediate product team. The kind of math is usually related to market sizing (e.g. is this a good opportunity) or the impact of changes on a certain set of key metrics (e.g. even if this change is successful it doesn’t move the needle).

  3. The link to the ‘Actionable Vs Vanity metrics’ is incorrect. Has an extra ‘http’

    Article is very useful from the other side of the table top 🙂

  4. Thank you Andy, great article!!!

  5. Andy. I do recommend your articles for the ones preparing for product management interviews and just to grab a hold on how products shall. Thanks for assimilating all the information and putting in a one stop shop for us 🙂

Speak Your Mind

*