Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6721

IllegalStateException when connecting to MongoDB using spark-submit

    XMLWordPrintableJSON

    Details

      Description

      I get the following exception when using saveAsNewAPIHadoopFile:

      15/03/23 17:05:34 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 4, 10.0.2.15): java.lang.IllegalStateException: open
      at org.bson.util.Assertions.isTrue(Assertions.java:36)
      at com.mongodb.DBTCPConnector.getPrimaryPort(DBTCPConnector.java:406)
      at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:184)
      at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:167)
      at com.mongodb.DBCollection.insert(DBCollection.java:161)
      at com.mongodb.DBCollection.insert(DBCollection.java:107)
      at com.mongodb.DBCollection.save(DBCollection.java:1049)
      at com.mongodb.DBCollection.save(DBCollection.java:1014)
      at com.mongodb.hadoop.output.MongoRecordWriter.write(MongoRecordWriter.java:105)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$12.apply(PairRDDFunctions.scala:1000)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$12.apply(PairRDDFunctions.scala:979)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
      at org.apache.spark.scheduler.Task.run(Task.scala:64)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      

      Before Spark 1.3.0 this would result in the application crashing, but now the data just remains unprocessed.

      There is no "close" instruction at any part of the code.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lrtrejo Luis Rodríguez Trejo
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: