Netflix Prize Winners May Be Disqualified
I’ve been following the Netflix Prize for years. This is the contest that gives all comers a chance to try and beat the formidable Cinematch algorithm that Netflix uses to give customers highly targeted recommendations. According to a recent email, Netflix may be preparing to announce the disqualification of the leading team “BellKor’s Pragmatic Chaos” who submitted results that beats Cinematch by the contest’s goal of 10%.
Due to my past entries in the contest itself, i’m on the mailing list to receive notices and announcements for participants. As of about 30 minutes ago, I received a general compliance email announcing that some participants have failed to comply with the Netflix Prize rules by creating multiple teams with an identical set of members. According to the Netflix rules, any participants and all teams to which they belong may be suspended from participation in the contest and may be ineligible for the coveted one million dollar prize.
Sounds fair right? So here is the rub. On June 26, 2009 Team “BellKor’s Pragmatic Chaos” reached the 10% improvement over the Cinematch algorithm required to claim the million dollar prize. According to the rules, once this goal is reached, all teams have 30 days to submit their final results to be considered for the prize. The clock is ticking and the last day of the contest is July 26, 2009. But…
BellKor’s Pragmatic Chaos is an amalgam of multiple teams that have collaborated to win the prize. The current leaderboard for the list looks like someone was playing musical chairs with the team names including things like “Pragmatic Theory”, “BellKor in BigChaos”, “BigChaos”, “BellKor” and so on.
The leading team openly admits that they have collaborated to create a winning algorithm and it may well be that the prestige of beating Cinematch will be sufficient to satisfy their competitive spirits but it sure seems like Netflix is already prepping the world for the announcement that the winning team may be disqualified.
I realize the rules were published years ago and have been very clearly stated but I have to admit, this sure feels like bad sportsmanship to me in a game that Netflix created and could benefit from since, also according to the rules, they own the submissions and the algorithms used to generate them.
admin
Update: here is a link to the announcement by Netflix http://tr.im/ruCZ
Chris Volinsky
I am a member of the BellKor’s Pragmatic Chaos team. We have not broken any rules, and no one has accused us as such. Each of the teams mentioned above are legitamate instantiations of teams, with different sets of team members, and are totally compliant with the rules. The history of the competition (as you well know) has been filled with independent teams combining to improve RMSE. In fact, this has been one of the wonderful discoveries in the competition, that blending teams can lead to substantial gains in RMSE. Perhaps you can explain why you think technical collaboration is bad sportsmanship? These partnerships are both within the letter and the spirit of the rules of the competition.
Netflix’ announcement today was to combat recent attempts to auto-register teams to submit multiple entries. See e.g. http://www.netflixprize.com//community/viewtopic.php?id=1466
admin
Chris,
I don’t think technical collaboration is bad sportsmanship. I think setting up rules in a way to avoid paying a hard earned prize is bad sportsmanship. I’m anticipating this to be the opening salvo in an attempt to avoid paying you guys the $1m you’ve obviously earned. So the question is this…have you or any member of your collaboration ever had more than one account with exactly the same team members. By definition that disqualifies you and every team of which that person is a member…
admin
Regarding the auto-register comment, I hope you are right. I’ve been following you and your team’s progress for over two years and you definitely deserve it. I would very much like to see you win this prize.
Chris Volinsky
Oh, OK…I think I mis-understood your accusation. There is no reason to believe that Netflix has acted in poor faith or would in any way avoid paying the prize. They have been excellent stewards of the competition so far and have been more than fair every step of the way.
By the way, it is *not* true that Netflix owns the winning algorithms. Those algorithms are licensed to Netflix for them to use as they see fit, but ownership is retained by the winner.
And yes, all of our teams are in compliance with the rules.
admin
On the issue of ownership, I stand corrected. And i’m actually quite a fan of Netflix but the timing of the announcement is a little inconvenient and a little suspicious. None of the auto-registration submissions to which you refer are actually qualifying submissions and don’t really seem like a threat to BellKor’s submission. I know there are a lot of hopefuls but it honestly seems as if Netflix only has to concern itself with whether BellKor’s Pragmatic Chaos has won the prize…or not.
For my part, I sincerely hope Netflix does the honorable thing and recognizes your efforts. Good luck Chris!