[SPARK-27216] Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.0, 2.2.0, 2.3.3, 2.4.0, 3.0.0
• Fix Version/s: 2.3.4, 2.4.2, 3.0.0
• Component/s: Spark Core
• Labels: correctness
• Assignee: Lantao Jin
• Reporter: Lantao Jin
• Created: 20/Mar/19 13:07
• Updated: 16/Oct/21 15:43
• Resolved: 04/Apr/19 01:11

Description

HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.

We can use below UT to reproduce:

  test("kryo serialization with RoaringBitmap") {
    val bitmap = new RoaringBitmap
    bitmap.add(1787)

    val safeSer = new KryoSerializer(conf).newInstance()
    val bitmap2 : RoaringBitmap = safeSer.deserialize(safeSer.serialize(bitmap))
    assert(bitmap2.equals(bitmap))

    conf.set("spark.kryo.unsafe", "true")
    val unsafeSer = new KryoSerializer(conf).newInstance()
    val bitmap3 : RoaringBitmap = unsafeSer.deserialize(unsafeSer.serialize(bitmap))
    assert(bitmap3.equals(bitmap)) // this will fail
  }

Upgrade to latest version 0.7.45 to fix it

Issue Links

• is duplicated by: SPARK-27530 FetchFailedException: Received a zero-size buffer for block shuffle
• is related to: SPARK-23178 Kryo Unsafe problems with count distinct from cache
• relates to: SPARK-24160 ShuffleBlockFetcherIterator should fail if it receives zero-size blocks
• relates to: SPARK-27367 Faster RoaringBitmap Serialization with v0.8.0
• relates to: SPARK-27530 FetchFailedException: Received a zero-size buffer for block shuffle
• links to: GitHub Pull Request #24157
• links to: GitHub Pull Request #24264
• links to: GitHub Pull Request #24290
• links to: GitHub Pull Request #24291