[NUTCH-2996] Use new SimpleRobotRulesParser API entry point (crawler-commons 1.4)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.20
• Fix Version/s: 1.20
• Component/s: robots
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 14/Aug/23 15:42
• Updated: 13/Mar/24 14:50
• Resolved: 22/Aug/23 08:46

Description

Crawler-commons 1.4 (#1085) robots.txt parser (SimpleRobotRulesParser) introduces a new API entry point to parse the robots.txt content:

• it's more efficient by accepting a collection of lower-cased, single-word user-agent product tokens, without the need to tokenize a (comma-separated) list of user-agent strings again with every robots.txt
• user-agent matching is compliant with RFC 9309 (section 2.2.1) only if the new API method is used

Issue Links

• Dependency: NUTCH-2995 Upgrade to crawler-commons 1.4
• links to: GitHub Pull Request #766