[NUTCH-3011] HttpRobotRulesParser: handle HTTP 429 Too Many Requests same as server errors (HTTP 5xx)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.19
• Fix Version/s: 1.20
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 01/Oct/23 12:07
• Updated: 13/Mar/24 14:51
• Resolved: 21/Oct/23 14:20

Description

HttpRobotRulesParser should handle HTTP 429 Too Many Requests same as server errors (HTTP 5xx), that is if configured signalize Fetcher to delay requests. See also NUTCH-2573 and https://support.google.com/webmasters/answer/9679690#robots_details

Issue Links

• links to: GitHub Pull Request #786