[NUTCH-98] RobotRulesParser interprets robots.txt incorrectly

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 0.7
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Jeff Bowden
• Created: 29/Sep/05 15:27
• Updated: 01/Apr/11 14:56
• Resolved: 01/Apr/11 14:56

Description

Here's a simple example that the current RobotRulesParser gets wrong:

User-agent: *
Disallow: /
Allow: /rss

The problem is that the isAllowed function takes the first rule that matches and incorrectly decides that URLs starting with "/rss" are Disallowed. The correct algorithm is to take the longest rule that matches. I will attach a patch that fixes this.