AI Summit 2017-09-25

Using machine learning

to improve matching in
marketplaces
About me

● Director of Engineering
for Airbnb’s Homes
business.
● Worked at Google for 6
years on Search.

Twitter: @surbs
Talk Outline

● Mapping out the user journey

● Search ranking
● Power of combining data and product changes
Mapping out the user
journey
Marketplace Booking Model

Search Click Buy

Marketplace Booking Model

Shopping
Search Click Buy
cart
Airbnb guest’s journey
Diverse Inventory
The mountain
house
Both the guest
and host have
preferences
Challenges - traditional vs. marketplace search

● Decision making is a long and sequential process

○ Average user does dozens of searches.
● No absolute and perfect relevance
○ Heterogeneous inventory and preferences.
Challenges - Airbnb vs. other marketplaces

● Two-sided market
○ Both the guest and host have preferences.
● Higher bar for commitment
○ Lots of money, vacation days and other travel dependencies. Your actual
experience is when you’re on the trip, and you can’t “return” that.
Airbnb Search to Booking Model

Search Click Book

Booking
Reject
Request

Accept
Search Ranking at
Airbnb
Modeling Ranking

● Regression rejected
○ f(query, listing) = utility -β
● Training
○ Use log data
○ Attribute outcomes viewed
γ << α
○ Assign empirical utilities

contacted
α

no action
0
Is training data representative of the user journey?
Winner’s trail

1 0 1 1 1
 ● Utilities were somewhat arbitrary.
● Utility is one level removed from user actions.
E.g. a predicted score of 0 could mean guest
hasn’t seen the result or that their booking isn’t
Shortcomings likely to go through or the guest won’t click on
it.
● It was hard to reason about the model
because of both guest and host side actions in
the same model.
Learning to Rank?
● Regression is a good starting point.

● But order is all the user sees, so why not optimize it directly?

● Well established literature from web search.

○ Evolution of pointwise to pairwise (RankNet) to weighted pairwise
(LambdaRank)
Pointwise vs. pairwise
● Pointwise
○ Score = F(listing | query)
○ Order based on scores
● Pairwise
○ Look at a pair of listings.
○ Order them based on best ranking
○ Minimizing the loss of bad ranking
Rank
Rank pairwise
preferred preferred

● Key idea:
○ Weight top position more in
pairwise loss.
 Training data
● Take a single search. Each row row contains 10 listings. One positive
example and nine negative examples for a single search that resulted in a
booking.

Search Result

Listing 0 Listing 1 Listing 2 Listing N

Booked listing
 Scoring

● Compute ranking and NDCG.

Listing 1 Listing 4 Listing 0 Listing 11

Positive sample Negative sample Negative sample

 Scoring

● Compute pairwise loss.

● Weigh the loss for each pair by the difference in the NDCG (LambdaRank).

Pairwise loss weighted by NDCG difference

Listing 9 Listing 4 Listing 1 Listing 0

Limitations of Rank
● Observations from experiments:
○ Good: guests contact more.
○ Bad: hosts acceptance decreases, more bad outcomes.
● Question:
○ Why does host acceptance drop? What went wrong?
● Hypothesis:
○ Lambda Rank focus only on pushing good outcomes to the top.
○ At Airbnb, it is equally important to push bad outcomes to the bottom.

● New Idea:
○ Weight the pairs differently
Tweaking Rank: Dual Discount Curves

Green: Good Outcomes

Red: Bad Outcomes
Modeling Ranking: What we learnt
● Build your data carefully
○ Label attribution
○ Data selection

● Formulate your model closer to real world.

○ Design loss function
■ Airbnb : Regression => Lambda => Lambda with tweaks
Power of combining
data and product
changes
Price trends
More confident decision
makers.