Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5949

Driver program has to register roaring bitmap classes used by spark with Kryo when number of partitions is greater than 2000

    XMLWordPrintableJSON

    Details

      Description

      When more than 2000 partitions are being used with Kryo, the following classes need to be registered by driver program:

      • org.apache.spark.scheduler.HighlyCompressedMapStatus
      • org.roaringbitmap.RoaringBitmap
      • org.roaringbitmap.RoaringArray
      • org.roaringbitmap.ArrayContainer
      • org.roaringbitmap.RoaringArray$Element
      • org.roaringbitmap.RoaringArray$Element[]
      • short[]

      Our project doesn't have dependency on roaring bitmap and HighlyCompressedMapStatus is intended for internal spark usage. Spark should take care of this registration when Kryo is used.

        Attachments

          Activity

            People

            • Assignee:
              irashid Imran Rashid
              Reporter:
              ptorok Peter Torok
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: