[NUTCH-2808] Document side effects of ignoring robots.txt

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Fix Version/s: 1.19
• Component/s: documentation, robots
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 11/Jul/20 12:05
• Updated: 13/Mar/24 14:50
• Resolved: 17/Dec/21 08:57

Description

(see NUTCH-1927 and NUTCH-2803)

The aim of NUTCH-1927 was to make it possible to ignore the robots.txt for a defined set of hosts/domains. Ignoring the robots.txt entirely has some site effects which should be documented:

• undesired content (duplicates, private pages, etc.) may get indexed
• the Crawl-Delay is ignored
• no sitemaps are detected (cf. NUTCH-2807)

Issue Links

• links to: GitHub Pull Request #711