Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27216

Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0, 2.1.0, 2.2.0, 2.3.3, 2.4.0, 3.0.0
    • 2.3.4, 2.4.2, 3.0.0
    • Spark Core

    Description

      HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.

      We can use below UT to reproduce:

        test("kryo serialization with RoaringBitmap") {
          val bitmap = new RoaringBitmap
          bitmap.add(1787)
      
          val safeSer = new KryoSerializer(conf).newInstance()
          val bitmap2 : RoaringBitmap = safeSer.deserialize(safeSer.serialize(bitmap))
          assert(bitmap2.equals(bitmap))
      
          conf.set("spark.kryo.unsafe", "true")
          val unsafeSer = new KryoSerializer(conf).newInstance()
          val bitmap3 : RoaringBitmap = unsafeSer.deserialize(unsafeSer.serialize(bitmap))
          assert(bitmap3.equals(bitmap)) // this will fail
        }
      

      Upgrade to latest version 0.7.45 to fix it

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cltlfcjin Lantao Jin
            cltlfcjin Lantao Jin
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment