[NUTCH-2598] URLNormalizerChecker fails on invalid URLs in input

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.14
• Fix Version/s: 1.16
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 13/Jun/18 16:18
• Updated: 28/Jan/21 13:56
• Resolved: 01/Sep/19 10:06

Description

I use the URLNormalizerChecker (urlnormalizer-regex and urlnormalizer-basic) to normalize URLs before further processing them. If one of the used normalizers throws a MalformedURLException when the URLNormalizer.normalize(...) method is called, this isn't caught and causes the checker to exit:

Exception in thread "main" java.net.MalformedURLException: For input string: "???120810002"
        at java.net.URL.<init>(URL.java:627)
        at java.net.URL.<init>(URL.java:490)
        at java.net.URL.<init>(URL.java:439)
        at org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer.normalize(BasicURLNormalizer.java:100)
        at org.apache.nutch.net.URLNormalizers.normalize(URLNormalizers.java:319)
        at org.apache.nutch.net.URLNormalizerChecker.process(URLNormalizerChecker.java:75)
        at org.apache.nutch.util.AbstractChecker.processStdin(AbstractChecker.java:97)
        at org.apache.nutch.util.AbstractChecker.run(AbstractChecker.java:77)
        at org.apache.nutch.net.URLNormalizerChecker.run(URLNormalizerChecker.java:71)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.nutch.net.URLNormalizerChecker.main(URLNormalizerChecker.java:80)
Caused by: java.lang.NumberFormatException: For input string: "???120810002"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:580)
        at java.lang.Integer.parseInt(Integer.java:615)
        at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:222)
        at java.net.URL.<init>(URL.java:622)
        ... 10 more

The URLNormalizer interface declares the MalformedURLException, it should be caught in the normalizer checker:

• log the error
• return/output empty string

Issue Links

• links to: GitHub Pull Request #435