I’ve worked on lots of recommender systems over the years and one of the most common questions that I have been asked by non-recommendery folk is, “But how exactly does the recommender algorithm work?”. It’s a question that I’ve come to loath, fear and love, in ever changing quantities.
I loath it because explaining the inner workings of most complex systems is a pain; it’s just hard to do. I often tend towards the abstract, since the details change too often in quickly evolving live systems, but then you don’t answer the ‘exactly’ part in the question. I fear it when the people asking want to build their own and are looking for a simple recipe to follow to arrive at destination Success. They often put too much emphasis on the algorithm and miss the product as a whole. Algorithms should be chosen to meet user/product needs and no one algorithm works for all. I love it because the people who ask are often genuinely interested and, when you love your job, who doesn’t love talking about it even more?
So I’ve started to answer the question “But how exactly does the recommender algorithm work?” by explaining what it’s meant to do. Talk about people who use it and how it tries to address their needs. Throw in a few stories about how difficult such problems are to solve, even for humans, let alone for machines who have limited interactions with users and a rather different experience of the world than we do. Point out that users are moving targets with interests and contexts that change over time, sometimes gradually, sometimes from one day to the next. Talk about the product as a whole, putting the algorithm in its proper context, as a single component in a larger system. Answering the question this way is making me loath and fear it a little less and love it a lot more.