[NUTCH-2807] SitemapProcessor to warn that ignoring robots.txt affects detection of sitemaps

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Fix Version/s: 1.19
• Component/s: robots, sitemap
• Labels: easytask
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 11/Jul/20 12:04
• Updated: 13/Mar/24 14:50
• Resolved: 17/Dec/21 08:59

Description

Ignoring the robots.txt causes as a site effect that no sitemaps can be detected via robots.txt.

SitemapProcessor should log a warning if robots.txt is ignored by configuration (NUTCH-1927/NUTCH-2803).

Issue Links

• links to: GitHub Pull Request #710