[NUTCH-2930] Protocol-okhttp: implement IP filter

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.18
• Fix Version/s: 1.19
• Component/s: plugin, protocol, security
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 11/Jan/22 15:02
• Updated: 13/Mar/24 14:51
• Resolved: 19/Aug/22 13:26

Description

In order to avoid information leakage to a public search index or web archive, it should be possible to configure Nutch in a way that no content is fetched from localhost, loop-back addresses, private address spaces.

NUTCH-2527 adds the configuration snippets to exclude URLs pointing to private addresses.

However, filtering URLs isn't enough because a DNS entry of an arbitrary host name may point to a private IP address. Blocking must happen on the protocol level because the IP address is only know in the protocol implementation. I'll add an implementation for protocol-okhttp.

Issue Links

• links to: GitHub Pull Request #736