Make Impactful Changes, Every

Time
Using Data to Hone Your
Mobile Product Focus

#mobileprodu
 Poll
• What’s your most important KPI(s) on mobile right now?
A) Downloads
B) DAUs/MAUs
C) LTV of Users
D) Visits
E) Ratings
F) Other: Type in Question Box

#mobileprodu
 Who Are We?
• Kendrick Wang • Emily Carrion

Product Marketing @ Head of Marketing

Apptimize @Apptentive
#mobileprodu
 Apptimize
Fast, intelligent mobile iterations, without disrupting the user
experience.

• Mobile A/B Testing

• Instant Update
• Feature Flagging
• Staged Rollouts

#mobileprodu
 Apptentive
Mobile customer engagement software to help enterprise brands
listen to, engage with and retain their mobile customers.

• Make your brand accessible

• Proactively engage at smart times
• Make data driven decisions

#mobileprodu
 Agenda
• You’ll Learn How To:
– Make smart product decisions
– Get qualitative and quantitative data before deploying
to all your users
– Validate their effects
– Make the most of limited resources

#mobileprodu
You’re Always Juggling Priorities
• Only have so many:
– Engineers
– Sprints
• Not everything can be on
the roadmap

#mobileprodu
 Everything Has a Cost
• There are always opportunity costs
• Lots of pressure on product decisions

#mobileprodu
You Have to Learn From All
Teams

#mobileprodu
 And Everyone Has An Input
“I have this idea…”
“What if we…”
“Our competitors have it...”
“Loads of users want to...”
“The next big thing...”
“It’ll only take a few minutes...”
#mobileprodu
Have you every had an executive
tell you to build a feature that you
didn’t think your users actually
wanted?

#mobileprodu
 Customer Feedback Justifies Product
Roadmap
PROBLEM
• Unjustified feature request led to
unhappy customers & 1-star ratings

SOLUTION
• Implemented Apptentive in-app
surveys

#mobileprodu
Customer Feedback Justifies Product Roadmap

RESULTS
• Within a week, collected 10,000
responses
• Used customer data to back up
product roadmap, and build better
app
• Rewarded with high volume of 5-star
ratings, and reduced churn

#mobileprodu
 Question:
• What other indicators or KPIs do you use
to determine what gets on the product
roadmap?
(use the question box)

#mobileprodu
 The Key Is In Data
• How can we be more deliberate about it?
– How do we :
• Determine the right data to look at?
• Prioritize the right data?
• Make sure the changes actually work?

#mobileprodu
What Do Most PMs Care
About?

#mobileprodu
Bonus: 16 Mobile Metrics that Matter
http://bit.ly/AdvMobilePM

#mobileprodu
 Understand How Users Behave

Qualitative Quantitative
Observation User Feedback Analytics
User Testing
Testing User Testing A/B Testing

#mobileprodu
 Conversion Funnels
• Where are the drop-off
points?
• When are users not
behaving how you
expect/want?

#mobileprodu
Conversion Funnels

#mobileprodu
 User Testing
• Hire testers to observe behavior
• Identify:
– Confusion
– Drop off points
– Unexpected behaviors
• Debrief + follow up
#mobileprodu
What if you could ask your actual
customers for feedback directly
in-app?

#mobileprodu
Customer Feedback – Your Secret Weapon

#mobileprodu
Use Customer Feedback to Release a New Product

#mobileprodu
Using these techniques helps you narrow
your focus

#mobileprodu
But you want to know a change’s effects
before you deploy to your entire
userbase

(There are no take backs in mobile)

#mobileprodu
Why can’t I browse titles before I sign
up?

#mobileprodu
#mobileprodu
So They Created a Hypothesis

#mobileprodu
Determine New User Behavior

#mobileprodu
 Netflix Findings
• You need to validate your ideas
• Back up your claims with data

#mobileprodu
 ‘s

Q: “What types of things does Netflix A/B Test?”

“Short answer – almost everything” –Neil Hunt, Chief Product Officer @

Netflix
#mobileprodu
 Deploy Without Disrupting
• Push winners out instantly
• Roll back changes instantly with feature flagging

#mobileprodu
 Most Effective Way to Test
• Be like Netflix
• Use A/B Testing
– Draws direct cause-effect relationship
– Gives you hard numbers
– Reveals assumptions

#mobileprodu
 Understand How Users Behave

Qualitative Quantitative
Observation User Feedback Analytics
User Testing
Testing User Testing A/B Testing

#mobileprodu
 Summary
Customer Feedback + A|B Testing
= Better Product

#mobileprodu
#CustomerLove

#mobileprodu
Questions?

#mobileprodu