Today’s digital-age consumers face an avalanche of product choices at every turn. Netflix has over 17,000 titles, Etsy has over 1.5 million registered sellers, Amazon sells over 480 million products. Finding the right choice within this overwhelming volume can feel like finding the right grain of sand on a beach.
As a result, consumers are increasingly reliant on product recommendations to guide their purchase decisions. Many consumers today buy products based on recommendations from people or brands they trust. Some of those recommendations are true word-of-mouth, but many online marketplaces use automated systems to make product recommendations, and those systems are inevitably based on a set of rules.
However, rule-based recommendation systems are only as effective as the person setting the rules. These systems rely heavily on administrative users to set and update the factors and thresholds that create recommendations. As product catalogs grow or user diversity increases, human intervention becomes a tedious—and often inaccurate—way to handle product recommendation.
To shift the burden away from human administrators, many companies have started implementing machine learning technology to streamline and automate the process of updating the rules that govern recommendation engines. One technique for applying machine learning to recommendation systems is the “collaborative filtering” method.
What is Collaborative Filtering?
Collaborative filtering started gaining momentum in the mid-2000s. That’s when Amazon launched its new machine-learning-powered recommendation engine, and Netflix gained notoriety for using complex learning algorithms to look beyond simple genre-based preferences when predicting what individual users would enjoy. Both these brands use multiple layers of collaborative filtering to continually improve the accuracy of their recommendations. The result is that, even with enormous product catalogs, the consumer experience is intuitive and not overwhelming.
The power behind these recommendation engines is Collaborative filtering. Techopedia defines collaborative filtering in the following way:
Collaborative filtering (CF) is a technique commonly used to build personalized recommendations on the Web. In collaborative filtering, algorithms are used to make automatic predictions about a user’s interests by compiling preferences from several users. Some popular websites that make use of the collaborative filtering technology include Amazon, Netflix, iTunes, IMDB, LastFM, Delicious and StumbleUpon. (source: techopedia)
Collaborative Filtering automatically makes predictions for how likely a user is to be interested in any item, based on his or her previous behavior and similarities to other users.
Here’s an example of the kind of logic that goes into CF:
If Tom and Sue both liked products A and B in the past and Sue also likes C, it is more likely that Tom will like C as too.
Now multiply that decision across millions of users and billions of data points, and you can see how powerful (and statistically accurate) these recommendations could get.
For more information on how this logic is applied at a large scale, check out this piece on Understanding semantic analysis.
Why are Collaborative Filtering Based Recommendation Engines So Important?
Let’s say you are a business that started off with just 100 products to offer. Your users will have a relatively small catalog to pick from. A simple recommendation system based on popularity or star ratings might work just fine for you. Because you have so few products, the system has a higher chance of recommending the best one based on this one simple input.
But, as product offerings and diversity increases (say, from 100 products to 10,000), the likelihood of accurately choosing the best product recommendation goes down significantly (there are likely to be more similar products in the catalog). Implementing such a basic recommendation engine would give less accurate recommendations, which can hurt sales and erode trust in your brand.
Collaborative filtering enables you to continually refine your recommendation process, ensuring that the accuracy of your recommendations is always improving, even as you add to the catalog.
In order to recommend products to your users, you need two things: an understanding of the users and an understanding of the products. Building a correlation between these two radically distinct entities is out of bounds for a basic recommendation system. Collaborative filtering based systems can actually create relationship maps between users and products by analyzing various purchase habits, items purchased, age, gender, other demographics and comparing them to a product’s performance (how often is it browsed/clicked/purchased and by whom).
Because collaborative filtering systems are based on the users themselves, they create a mutually beneficial equation:
Customers discover more products or content that they like and experience less frustration when approaching a large catalog.
Brands get increased customer engagement and revenue, and they also see increases in customer retention and loyalty as they have become a trusted advisor in the purchase/consumption process.
What could better recommendations mean for your business?