To distract me from my ever approaching 1st year exams I decided to create a site that recommends people films to watch based upon their previous viewing habits. I gave it to my girlfriend to use and evaluate and she gave positive feedback - we even found and enjoyed a few films through it that we would never have found without it. As with most things technological the how is better than the why and what, so in this post I will talk a bit about its internals and specifically how it works by cheating.
The site is pretty cool. You sign in using Facebook (or sign up using your email) and you will be taken to a page where you can search for previous films you have watched. Simply type in a name of a film you have liked and press the green thumbs up next to its name to add it to the list of films you have enjoyed. Once you have added a few (10+) films click on the "Show me my recommendations" button and you will be taken to a page listing films the site thinks you will like. For each recommendation you will get a film poster, a short description and some other metadata about it (like who stars in it, its ratings and language). You can watch the youtube trailer for it by clicking the blue film icon, or if you have seen it you can add it to your watch list with the green thumbs up, and conversely if you have seen it and disliked it click the red thumbs down to hide the film forever. Click the plus button if you are interested in seeing it to save it for watching later.
The site cheats and uses the IMDB's static search files hosted on S3 meaning it doesn't need to have a complete movie dataset. When a user says they have watched a film the site cheats and pulls the metadata from IMDB using a task queue into the database along with all the films the IMDB recommends. When a user wishes to see his recommendations the most recommended films out of all the films he has watched and liked are displayed.
The site itself is a mix of technologies: its written in Python using the Flask micro-framework, served by a few gunicorn instances sitting behind nginx. It uses PostgreSQL to store the data and celery to pull jobs from a RabbitMQ task queue. The frontend uses Backbone.js to power it (and this was my first experience using it - a positive one).
The real trickery starts when a user "likes" a film from the search page. When this happens the film appears instantly in the "likes" section, but really a message is sent to the server with the IMDB ID of the film and the software checks the database. If a film with that IMDB is not in it then a task is dispatched into the queue for processing, and at a later point (mostly nearly instantly) the task workers pick up the IMDB id and go fetch the film info from the IMDB page using IMDBpy and add the metadata to the database.
After we have the metadata about the film the user wants to watch we need to compute what other films he might enjoy. I could spend millions on creating my own ranking algorithm but I don't have nearly as many datapoints or resources as Netflix or Amazon. So lets cheat. The site pulls all the recommendations from the IMDB page as well, and then launches new task's to pull the info about each of the films if they don't already exist in the database.
This whole process normally takes only a few seconds to complete so is mostly transparent to the user, because by the time they have loaded the recommendations page most, if not all of the recommendations have been fetched. The ranking algorithm for the recommendations is kind of cheating as well: we simply grab the recommendations from all of the films the user likes from the database and rank them by the number of times they appear - i.e films that are recommended the most times by the films the user has liked appear at the top and those recommended the least appear at the bottom. Its a little more complex than that when it comes to the filtering rules the user has applied, but that's pretty much the jist.
The frontend isn't very interesting. I decided to give backbone.js a spin for this project and I sort of regret it. Don't get me wrong, Backbone.js is pretty good but I found the lack of tutorials and samples hindered my progress. Using Backbone.js to model the data is pretty cool - because it only interacts with the server via AJAX requests to pull and push JSON-encoded data I could (if I was crazy) re-write the entire thing in C#, PHP or Ruby without changing the frontend. Its also a lot more efficient than rendering the whole page server-side and pushing it to the client which means the interface feels more responsive and the server load stays low.
So... yeah. I'm always bad at ending blog posts, so go away.