A search engine that learns from your behaviour

In recent decades, search engines such as Google, Bing and DuckDuckGo have become indispensable instruments for finding information in the continually expanding digital universe. As such, it is has become increasingly important that the search results found tie in with what the user is actually searching for.

Researchers who develop search algorithms dream of a self-learning, personalised search engine; one that learns to adapt to what an individual user wants. Between 2016 and 2021, professor Maarten de Rijke from the University of Amsterdam led an NWO research project that aimed to contribute to the development of such a self-learning, personalised search engine: project CLeaR, the acronym for Contextual Learning to Rank.

How does a self-learning, personalised search engine differ from the search engine before the project started? ‘What is truly useful for a user depends, amongst other things, on the user’s knowledge and experience and they have previously searched for’, De Rijke says. ‘It cannot simply be captured by merely examining the contents of documents on the web.’

For a long time, examining the contents of web pages was the basis of each search algorithm, and that was continuously improved over the years, of course. In 1998, Google was the first to use a search algorithm that not only examined how often a certain word occurred on a webpage, which all other search engines already did, but also took into account how often the page was referred to from other pages. This second search principle proved to be a golden find, which gave Google a major advantage.

However, De Rijke and his colleagues are also trying to implicitly include the personal aspect of the user. De Rijke: ‘Where does somebody click? What do they download? How long do they spend reading something? Based on the interaction between the user and the search engine, we are trying to understand how the search results can best tie in with the context, knowledge and need of the user.’

Learning from users

Harrie Oosterhuis was one of the three PhDs who worked on the project. What does he consider to be the most important outcome of his research? ‘We developed a statistical method that learns from the clicking behaviour of users’, Oosterhuis says. ‘If you show a list of results to users, most will only pay attention to the top two or three results. The position of a result can therefore lead to a significant advantage or disadvantage that has nothing to do with the relevance. You need to take this into account when comparing clicking behaviour on different results. For example, if you see that ten percent of the users click on the first result and only five percent on the tenth result, then that tenth result is probably far more attractive. Our method maps such incorrect advantages and disadvantages and corrects for them.’

You could say that the search results which wrongly end up lower on the list of results but which people definitely click on more often should get an extra boost upwards. Oosterhuis obtained his PhD cum laude and won a Best Paper Award for his research at the renowned WSDM conference of search engines in 2021.

Open research community

Oosterhuis saw that his research results were quickly picked up by fellow researchers from both academia and big tech. He does not know for certain whether the results are already used in commercial search engines such as Google, but the chances are high. ‘Another significant advantage of our method,’ Oosterhuis says, ‘is that it not only works when searching for webpages but also when searching for products, images, videos, emails or internal documents.’

Source: NWO