Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2598

URLNormalizerChecker fails on invalid URLs in input

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.14
    • Fix Version/s: 1.15
    • Component/s: None
    • Labels:
      None

      Description

      I use the URLNormalizerChecker (urlnormalizer-regex and urlnormalizer-basic) to normalize URLs before further processing them. If one of the used normalizers throws a MalformedURLException when the URLNormalizer.normalize(...) method is called, this isn't caught and causes the checker to exit:

      Exception in thread "main" java.net.MalformedURLException: For input string: "???120810002"
              at java.net.URL.<init>(URL.java:627)
              at java.net.URL.<init>(URL.java:490)
              at java.net.URL.<init>(URL.java:439)
              at org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer.normalize(BasicURLNormalizer.java:100)
              at org.apache.nutch.net.URLNormalizers.normalize(URLNormalizers.java:319)
              at org.apache.nutch.net.URLNormalizerChecker.process(URLNormalizerChecker.java:75)
              at org.apache.nutch.util.AbstractChecker.processStdin(AbstractChecker.java:97)
              at org.apache.nutch.util.AbstractChecker.run(AbstractChecker.java:77)
              at org.apache.nutch.net.URLNormalizerChecker.run(URLNormalizerChecker.java:71)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
              at org.apache.nutch.net.URLNormalizerChecker.main(URLNormalizerChecker.java:80)
      Caused by: java.lang.NumberFormatException: For input string: "???120810002"
              at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
              at java.lang.Integer.parseInt(Integer.java:580)
              at java.lang.Integer.parseInt(Integer.java:615)
              at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:222)
              at java.net.URL.<init>(URL.java:622)
              ... 10 more
      

      The URLNormalizer interface declares the MalformedURLException, it should be caught in the normalizer checker:

      • log the error
      • return/output empty string

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wastl-nagel Sebastian Nagel
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: