[NUTCH-1084] ReadDB url throws exception

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.3
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 19/Aug/11 16:56
• Updated: 18/Nov/16 10:10
• Resolved: 18/Nov/16 10:09

Description

Upgrade your Apache Hadoop cluster to 2.7.3 or higher to solve the problem.

Readdb -url suffers from two problems:
1. it trips over the _SUCCESS file generated by newer Hadoop version
2. throws can't find class: org.apache.nutch.protocol.ProtocolStatus (???)

The first problem can be remedied by not allowing the injector or updater to write the _SUCCESS file. Until now that's the solution implemented for similar issues. I've not been successful as to make the Hadoop readers simply skip the file.

The second issue seems a bit strange and did not happen on a local check out. I'm not yet sure whether this is a Hadoop issue or something being corrupt in the CrawlDB. Here's the stack trace:

Exception in thread "main" java.io.IOException: can't find class: org.apache.nutch.protocol.ProtocolStatus because org.apache.nutch.protocol.ProtocolStatus
        at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:204)
        at org.apache.hadoop.io.MapWritable.readFields(MapWritable.java:146)
        at org.apache.nutch.crawl.CrawlDatum.readFields(CrawlDatum.java:278)
        at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
        at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:524)
        at org.apache.hadoop.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:105)
        at org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:383)
        at org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:389)
        at org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:514)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Issue Links

• is related to: HADOOP-12406 AbstractMapWritable.readFields throws ClassNotFoundException with custom writables