Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2009

Fetcher does not work with batchID

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.3
    • Fix Version/s: 2.3.1
    • Component/s: fetcher
    • Labels:
      None
    • Environment:

      On Ubuntu 14.04 nutch 2.3 configured to use MongoDB for storage with gora-mongodb 0.5.

      The same issue did NOT occur with gora-hbase (HBase 0.90.14).

      Description

      The fetcher does only work with the option -all.
      It does not work when providing a batch-ID.

      $home/apache-nutch-2.3/runtime/local$ bin/nutch fetch 1431538082-1014788459
      FetcherJob: starting at 2015-05-13 19:28:30
      FetcherJob: batchId: 1431538082-1014788459
      FetcherJob: threads: 10
      FetcherJob: parsing: false
      FetcherJob: resuming: false
      FetcherJob : timelimit set for : -1
      java.lang.IllegalArgumentException: can't serialize class org.apache.avro.util.Utf8
      at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:284)
      at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:185)
      at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:131)
      at com.mongodb.DefaultDBEncoder.writeObject(DefaultDBEncoder.java:33)
      at com.mongodb.OutMessage.putObject(OutMessage.java:289)
      at com.mongodb.OutMessage.writeQuery(OutMessage.java:211)
      at com.mongodb.OutMessage.query(OutMessage.java:86)
      at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:81)
      at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
      at com.mongodb.DBCursor._check(DBCursor.java:458)
      at com.mongodb.DBCursor._hasNext(DBCursor.java:546)
      at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
      at org.apache.gora.mongodb.query.MongoDBResult.nextInner(MongoDBResult.java:69)
      at org.apache.gora.query.impl.ResultBase.next(ResultBase.java:114)
      at org.apache.gora.mapreduce.GoraRecordReader.nextKeyValue(GoraRecordReader.java:119)
      at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
      at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
      at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                colibri Saskia Vola
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: