Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13926

Automatically use Kryo serializer when shuffling RDDs with simple types

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: Shuffle, Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      Because ClassTags are available when constructing ShuffledRDD we can use them to automatically use Kryo for shuffle serialization when the RDD's types are guaranteed to be compatible with Kryo (e.g. RDDs whose key, value, and/or combiner types are primitives, arrays of primitives, or strings). This is likely to result in a large performance gain for many RDD API workloads.

        Attachments

          Activity

            People

            • Assignee:
              joshrosen Josh Rosen
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: