Netflix Prize Winners May Be Disqualified

Posted on the July 8th, 2009 under Editorial,Social,Software,Technology by Bryan Waters

I’ve been following the Netflix Prize for years.  This is the contest that gives all comers a chance to try and beat the formidable Cinematch algorithm that Netflix uses to give customers highly targeted recommendations.  According to a recent email, Netflix may be preparing to announce the disqualification of the leading team “BellKor’s Pragmatic Chaos” who submitted results that beats Cinematch by the contest’s goal of 10%.

Due to my past entries in the contest itself, i’m on the mailing list to receive notices and announcements for participants.  As of about 30 minutes ago, I received a general compliance email announcing that some participants have failed to comply with the Netflix Prize rules by creating multiple teams with an identical set of members.  According to the Netflix rules, any participants and all teams to which they belong may be suspended from participation in the contest and may be ineligible for the coveted one million dollar prize.

Sounds fair right?  So here is the rub.   On June 26, 2009 Team “BellKor’s Pragmatic Chaos” reached the 10% improvement over the Cinematch algorithm required to claim the million dollar prize.    According to the rules, once this goal is reached, all teams have 30 days to submit their final results to be considered for the prize.  The clock is ticking and the last day of the contest is July 26, 2009.   But…

BellKor’s Pragmatic Chaos is an amalgam of multiple teams that have collaborated to win the prize.  The current leaderboard for the list looks like someone was playing musical chairs with the team names including things like “Pragmatic Theory”, “BellKor in BigChaos”, “BigChaos”, “BellKor” and so on.

The leading team openly admits that they have collaborated to create a winning algorithm and it may well be that the prestige of beating Cinematch will be sufficient to satisfy their competitive spirits but it sure seems like Netflix is already prepping the world for the announcement that the winning team may be disqualified.

I realize the rules were published years ago and have been very clearly stated but I have to admit, this sure feels like bad sportsmanship to me in a game that Netflix created and could benefit from since, also according to the rules, they own the submissions and the algorithms used to generate them.

6 Responses to 'Netflix Prize Winners May Be Disqualified'

  1. July 8, 2009 at 6:36 pm
  2. July 8, 2009 at 7:18 pm
    Chris Volinsky
  3. July 8, 2009 at 7:21 pm
  4. July 8, 2009 at 7:26 pm
  5. July 8, 2009 at 7:31 pm
    Chris Volinsky
  6. July 8, 2009 at 7:41 pm

Leave a Reply

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>