Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1084

ReadDB url throws exception

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.3
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Upgrade your Apache Hadoop cluster to 2.7.3 or higher to solve the problem.

      Readdb -url suffers from two problems:
      1. it trips over the _SUCCESS file generated by newer Hadoop version
      2. throws can't find class: org.apache.nutch.protocol.ProtocolStatus (???)

      The first problem can be remedied by not allowing the injector or updater to write the _SUCCESS file. Until now that's the solution implemented for similar issues. I've not been successful as to make the Hadoop readers simply skip the file.

      The second issue seems a bit strange and did not happen on a local check out. I'm not yet sure whether this is a Hadoop issue or something being corrupt in the CrawlDB. Here's the stack trace:

      Exception in thread "main" java.io.IOException: can't find class: org.apache.nutch.protocol.ProtocolStatus because org.apache.nutch.protocol.ProtocolStatus
              at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:204)
              at org.apache.hadoop.io.MapWritable.readFields(MapWritable.java:146)
              at org.apache.nutch.crawl.CrawlDatum.readFields(CrawlDatum.java:278)
              at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
              at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:524)
              at org.apache.hadoop.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:105)
              at org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:383)
              at org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:389)
              at org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:514)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      

        Attachments

        1. NUTCH-1084.patch
          0.5 kB
          Julien Nioche

          Issue Links

            Activity

              People

              • Assignee:
                markus17 Markus Jelsma
                Reporter:
                markus17 Markus Jelsma
              • Votes:
                3 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: