Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2503

Lower shuffle output buffer (spark.shuffle.file.buffer.kb)

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Shuffle, Spark Core
    • Labels:
    • Target Version/s:

      Description

      Each shuffle output stream uses a 100KB buffer. This is simply too large. For 10k reducers, 32 cores, that adds up to 32GB in buffer alone.

      The change is simple, but we should make sure there isn't much performance regression when lowering the buffer size.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rxin Reynold Xin
                Reporter:
                rxin Reynold Xin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: