Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11016

Spark fails when running with a task that requires a more recent version of RoaringBitmaps

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.6.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      The following error appears during Kryo init whenever a more recent version (>0.5.0) of Roaring bitmaps is required by a job. org/roaringbitmap/RoaringArray$Element was removed in 0.5.0

      A needed class was not found. This could be due to an error in your runpath. Missing class: org/roaringbitmap/RoaringArray$Element
      java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
      	at org.apache.spark.serializer.KryoSerializer$.<init>(KryoSerializer.scala:338)
      	at org.apache.spark.serializer.KryoSerializer$.<clinit>(KryoSerializer.scala)
      	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
      	at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
      	at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:222)
      	at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
      	at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
      	at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
      	at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:85)
      	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
      	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
      	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
      	at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
      	at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
      	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
      	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
      	at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
      	at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
      	at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
      	at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
      	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
      	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
      	at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
      	at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
      

      See https://issues.apache.org/jira/browse/SPARK-5949 for related info

        Attachments

          Activity

            People

            • Assignee:
              drcrallen Charles R Allen
              Reporter:
              drcrallen Charles R Allen

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment