[NUTCH-992] SolrDedup is broken in 2.x

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: nutchgora
• Fix Version/s: 2.3
• Component/s: indexer
• Assignee: Unassigned
• Reporter: Markus Jelsma
• Created: 28/Apr/11 11:23
• Updated: 01/May/14 06:23
• Resolved: 25/Jun/13 20:21

Description

SolrDedup seems to have been broken for at least a few months, perhaps more. It does fetch the documents from Solr but when processing the rows we get the following exception:

Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:899)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
at org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:350)
at org.apache.nutch.indexer.solr.SolrDeleteDuplicates.run(SolrDeleteDuplicates.java:360)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.solr.SolrDeleteDuplicates.main(SolrDeleteDuplicates.java:370)

Issue Links

• is related to: NUTCH-1571 SolrInputSplit doesn't implement Writable and crawl script doesn't pass crawlId to generate and updatedb tasks