Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5915

Spillable should check every N bytes rather than every 32 elements

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 1.0.0
    • None
    • Shuffle, Spark Core

    Description

      Spillable currently checks for spill every 32 elements. However, this puts it at a risk of OOM if each element is large enough. A better alternative is to check every N bytes accumulated.

      N should be decided to a reasonable number via proper testing.

      This is a follow-up of SPARK-4808, and was discussed originally in https://github.com/apache/spark/pull/4420.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-4808 Spark fails to spill with small number of large objects

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              mkim Mingyu Kim
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: