Ranking in a two-sided matching platform (Part 1)

Aug 15, 2023 learning to rank uplift modelling matching market marketplace

Overview #

Profi is a two-sided matching platform where customers can find tutors, beauty masters, plumbers, and other professionals, and professionals can find customers. It's a horizontal marketplace with over 900 types of services provided by professionals.

The quality of search and ranking is important for the matching process there. In a two-sided matching platform, both sides have to choose each other for a deal to happen, and that makes the ranking problem harder and kind of unique.

In two blog posts, I'll explore the ranking problem and explain the solution that significantly increased the number of deals in the marketplace.

In this first post, I will cover:

• The concept of matching markets and their distinctions from other markets.
• The mechanics of Profi's matching process.
• Business objectives.
• The three versions of the problem formulation, with insights into why certain approach succeeded while others did not.

In the second post, I will cover the details of the successful solution that was implemented in production.

Matching markets #

When a customer places an order on Amazon, the seller doesn't get to choose whether or not to sell to this customer. And this is an example of a commodity market.

In matching markets, you can't just choose and get what you want—you also have to be chosen. Examples of matching markets and platforms:

• Labor market.
• Airbnb before introducing the instant booking feature.
• Dating platforms, like Tinder and Bumble.
• Platforms for freelancers, like Upwork and Fiverr.

While in commodity markets prices do most of the work and balance supply and demand, in matching markets, price is not the most important factor. Market design, search, and ranking algorithms play a crucial role there.

To learn more about matching markets read Who Gets What—and Why by Alvin Roth, a Nobel Prize laureate in Economics.

Matching process #

The process of how customers and professional can find each other on Profi:

• Customers place orders and describe the details of the work.
• Professionals search, choose orders, and send proposals to the customers.
• The customers review the proposals and choose the professionals, or don't choose anyone.
• If a professional sends a proposal to a customer, and the customer choses the professional then this is a deal.

Business objective #

The order feed in the app for professionals is the source of 56% of proposals. Orders in the feed are selected based on a professional's preferences and previous activity. Originally, orders were sorted in reverse chronological order.

While order "freshness" is important for professionals, it's not the only factor of choice. The hypothesis was that we could improve the matching process by implementing a new order ranking algorithm.

The objective was to increase the number of deals, which is one of the two most important metrics in the company.

Problem formulation #

Probability of a proposal #

The first idea was to sort orders by the probability of a proposal. The intuition was the following:

• Sorting by the probability of a proposal would help professionals to find more relevant orders.
• Professionals would send more proposals.
• Customers would receive more proposals and have more choice.
• A wider choice of proposals would increase the probability of finding the right professional.
• The number of deals would increase.

We trained a machine learning model with the following target and conducted an experiment:

The experiment resulted in the following:

• The number of proposals had indeed increased.
• But the number of deals had not increased.

The interpretation was that extra proposals were sent to customers who already had enough proposals to choose from.

Increasing the number of proposals without increasing the number of deals is not good both for professionals and for customers:

• Professionals make an extra effort to write and send proposals, but they don't get more customers.
• Customers are distracted by additional proposals while they already have enough to choose from.

Probability of a deal #

The next intuitive choice was to sort orders by the probability of a deal, and to train a model with the following target:

To be precise, for a pair of a professional and a customer (P, C), a model should've predicted the probability that:

• The professional P would send a proposal to the customer C.
• And the customer C would choose the professional P among others.

But, if you think carefully, you can see that this is not the right problem formulation for the case:

• First, imagine that the first professional on the picture above has sent a proposal to the first customer, and that this customer has chosen this professional.
• Then the second professional wouldn't get a deal with the first customer. Does that mean that the second professional shouldn't have sent the proposal?
• Now, imagine that the first professional hasn't sent a proposal. Then the proposal from the second professional could be valuable.

We decided not to move forward with this formulation of the problem as it doesn't correctly reflect the matching process.

Uplift in the probability of a deal #

The last idea was to estimate how showing an order to a professional would change the probability of a deal in that order, no matter with which professional. The model should calculate a difference in the probability of a deal in an order between two parallel universes:

• The one where we show the order to the professional.
• The one where we don't show the order to the professional.

Models that compare parallel universes are called uplift models.

Ranking orders by the uplift in the probability of a deal showed the following results in the experiment (measured with proxy metrics):

• The number of deals coming from the order feed had increased by 10%.
• The total number of deals had increased by 5%.

It was a great result in itself. But it also opened a door for further improvements, not only in ranking of orders. For example, finding more relevant orders and adding them to the order feed automatically increases the number of deals.

I will get into some details of the model in the next post. Stay tuned.

P.S. Type III error #

This case reminded me of a notion of the type III error. Before I explain what it means, let me give you examples of the type I and type II errors in the context of the ranking of orders:

• Imagine we show an order to a professional and expect that the professional to send a proposal. But the professional doesn't send a proposal. That's a type I error.
• Now imagine we don't show an order to a professional because we don't expect the professional to send a proposal. But if we showed it, the professional would send a proposal. That's a type II error.

Well, whether the professional would send a proposal or not, is not the right question in the first place, at least not the final one. And that's a type III error.

Also, check out Cassie Kozyrkov's excellent explanation: https://youtu.be/R51nueuAyIY.

© Evgeny Ivanov 2023