[NUTCH-1927] Create a whitelist of IPs/hostnames to allow skipping of RobotRules parsing

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10
• Component/s: fetcher
• Labels: available patch
• Assignee: Chris A. Mattmann
• Reporter: Chris A. Mattmann
• Created: 29/Jan/15 20:55
• Updated: 13/Mar/24 14:51
• Resolved: 18/Apr/15 16:35

Description

Based on discussion on the dev list, to use Nutch for some security research valid use cases (DDoS; DNS and other testing), I am going to create a patch that allows a whitelist:

<property>
  <name>robot.rules.whitelist</name>
  <value>132.54.99.22,hostname.apache.org,foo.jpl.nasa.gov</value>
  <description>Comma separated list of hostnames or IP addresses to ignore robot rules parsing for.
  </description>
</property>

Issue Links

• is required by: NUTCH-1992 Port whitelist from NUTCH-1927 to 2.x