[NUTCH-2859] urlnormalizer-protocol: allow to normalize domains

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.18
• Fix Version/s: 1.19
• Component/s: plugin, urlnormalizer
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 29/Mar/21 14:31
• Updated: 13/Mar/24 14:50
• Resolved: 06/Apr/21 14:53

Description

The plugin urlnormalizer-protocol normalizes the URL protocol/scheme for a given list of hosts to the desired "normal" protocol (usually one of http or https). It would be handy to allow to specify domain names as well, so that all hosts/subdomains in a given domain are normalized.

In order to stay backward-compatible this could be done by matching *.example.org as a pattern for all hosts or subdomains of the domain example.org.

Issue Links

• links to: GitHub Pull Request #576