Improving Content-Based Recommendation using Product Images


Using web analytics data we observed that a substantial number of our products have a small number of user events associated with them (where events are features such as product detail page visits, add to cart and successful orders). In Germany, less than 20% of our product catalog have some recommendations supported by Collaborative Filtering approximation. This leaves us with the other 80% exclusively based on Content-Based Filtering. However, previous tests have proven the Collaborative Filtering had higher conversion rate.

For the Content-based Recommender, we use article attributes as input features. These include properties such as brand, category, specified colour, dimensions, material, price, series, shape, style and supplier.

To measure the performance of this content-based recommender we did a simple web application which for a given product, provided the results for [1] the content-based recommender and [2] the Collaborative Filtering approach (for the sake of this article, let’s call this the Click Through Rate Recommender). This Click Through Rate Recommender could justify a blog post by itself and in very simple terms it calculates a position where a recommendation is presented as having a certain probability of being clicked. The score presented is the sum of the probabilities of a certain recommendation being clicked minus the effective number of clicks that the recommendation had. Positive values means the product has more clicks than expected and negative values means the product has less clicks than expected.

Looking at the example below, you can see that using the current set of input features for Content Based Filtering (Diagram A), it looks like these fields are not enough. More than 60% of the attributes match on the first results but the Click Through Rate Recommender (Diagram B) shows that the user prefers to click on products with mix wood and metal.


Diagram A – Content-Based Recommender output


Diagram B – Click Through Recommender output



Diagram C – Content-Based Recommender output

colour 3Diagram D – Click Through Recommender output

Colour 4

In this case, the mirror seems to be important for the user.


Diagram E – Content-Based Recommender output


Diagram F – Click Through Recommender output


In this particular example, the user probably considers the colour of the wood to be important in his selection criteria.


Based on the above outcome, we decided to explore photos as another input feature. From analysis using features extracted from the photos, we deduced that there is a strong correlation between material and colour. Hence, we extracted a small number of representative colours for each image (palette) and after compared these palettes.

Extract a small number of representative colours

While doing research we discovered that this is a problem which could be solved in various ways. This problem is usually known as Colour Quantization. After some research we decided to follow something similar to Colour Reduction Using K-Means Clustering to build our palette. We decide to go with these parameters:

  • 10-colour palette per image
  • In order to improve performance, we sampled 500 non-cleaned points on the image to build the palettes. We tested 100.000, 10.000, 5.000, 1.000, 100, 20 non-cleaned with no massive change on the results over 500 points. Evaluated with the square distance of a base clustering of 100.000 points.colour7
  • We calculate the centroids 3 times to reduce the probability of having a local minimum
  • After seeing the costs on each step (see next graph), 20 steps seem to be enough to learn the colours on the Clustering Algorithmcolour8

Palette examples

In order to understand what the colour extraction algorithm is producing, let’s visualise some palettes.

  • The first line is the product that we are basing the recommendations on,
  • the second is the best result of the Content-based Recommender and
  • the third is the best result of the Click-Through Recommender.

Blocks 1

In the example above, we find the mix of wood brown colours and metal grey colours.

Block 2

In the second example above, we can deduce that there is potential risk with the background where the photo is taken, the mirror will reflect the brown floor (which in this case is not so bad). Later we will see that having the open door with the dark interior has a great influence on the colour recommender as well.

blocks 3

In this last example above, the dark wood is clear on the first and third photo, the second one has lighter wood.


As an introduction, lets see the results of Colour Recommender alone when used on the previous examples.


  • For the first example and result, 2 results actually have the metal/wood combination. The recommender confuses the cover of the first table with the metal colour.
  • For the second example and result, one can observe 4 articles with mirrors, 4 of them are white, and the black colour comes from inside the closet.
  • For the third example and result, all them are dark wood.

Now hand picking some amazing results:


Considering the fact that this recommendation was done with just colour in an image and after a promising off-line evaluation, we decided to do an Interleave User Test

Results and Conclusion

The User Tests results were great. Resuming the numbers, we observed a Click-Through uplift of ~10% and a conversion rate uplift of ~4% which we could attribute to the variant with this feature improvement. For articles with low user behaviour, we found a Click-Through uplift of ~12%.

The fact that this was a simple change gives us reason to think that investing in further improving and testing with new features can lead to a better customer experience and higher conversion rates.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s