[NUTCH-2028] java.lang.IllegalArgumentException: can't serialize class org.apache.avro.util.Utf8

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3
• Fix Version/s: 2.3.1
• Component/s: fetcher

Environment

Mac 10.10.3 Nutch 2.3

• Assignee: Unassigned
• Reporter: Roman P
• Created: 31/May/15 16:33
• Updated: 13/Mar/24 14:51
• Resolved: 20/Sep/15 12:56

Description

Compiled Nutch 2.3 with MongoDB as a persistence. Getting exception when fetching. Searched for similar errors online, noticed that this issue was addressed in gora 0.6. Tried recompiling with 0.6 but then getting different exception, seems that it's incompatible with hadoop 1.2.0. Tried different versions of hadoop with no luck.

FetcherJob: starting at 2015-05-31 09:29:04
FetcherJob: batchId: all
FetcherJob: threads: 10
FetcherJob: parsing: false
FetcherJob: resuming: false
FetcherJob : timelimit set for : -1
java.lang.IllegalArgumentException: can't serialize class org.apache.avro.util.Utf8
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:284)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:185)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:131)
at com.mongodb.DefaultDBEncoder.writeObject(DefaultDBEncoder.java:33)
at com.mongodb.OutMessage.putObject(OutMessage.java:289)
at com.mongodb.OutMessage.writeQuery(OutMessage.java:211)
at com.mongodb.OutMessage.query(OutMessage.java:86)
at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:81)
at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
at com.mongodb.DBCursor._check(DBCursor.java:458)
at com.mongodb.DBCursor._hasNext(DBCursor.java:546)
at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
at org.apache.gora.mongodb.query.MongoDBResult.nextInner(MongoDBResult.java:69)
at org.apache.gora.query.impl.ResultBase.next(ResultBase.java:114)
at org.apache.gora.mapreduce.GoraRecordReader.nextKeyValue(GoraRecordReader.java:119)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)