[NUTCH-1032] Delegate parsing of robots.txt to crawler-commons

• Type: Task
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Fix Version/s: 1.4, nutchgora
• Labels: robots.txt
• Assignee: Julien Nioche
• Reporter: Julien Nioche
• Created: 06/Jul/11 13:35
• Updated: 20/Dec/11 11:30
• Resolved: 06/Jul/11 20:34

Description

We're about to release the first version of Crawler-Commons http://code.google.com/p/crawler-commons/ which contains a parser for robots.txt files. This parser should also be better than the one we currently have in Nutch. I will delegate this functionality to CC as soon as it is available publicly