Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19659

Fetch big blocks to disk when shuffle-read

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.2.0
    • Component/s: Shuffle, Spark Core
    • Labels:
      None

      Description

      Currently the whole block is fetched into memory(offheap by default) when shuffle-read. A block is defined by (shuffleId, mapId, reduceId). Thus it can be large when skew situations. If OOM happens during shuffle read, job will be killed and users will be notified to "Consider boosting spark.yarn.executor.memoryOverhead". Adjusting parameter and allocating more memory can resolve the OOM. However the approach is not perfectly suitable for production environment, especially for data warehouse.

      Using Spark SQL as data engine in warehouse, users hope to have a unified parameter(e.g. memory) but less resource wasted(resource is allocated but not used),

      It's not always easy to predict skew situations, when happen, it make sense to fetch remote blocks to disk for shuffle-read, rather than
      kill the job because of OOM. This approach is mentioned during the discussion in SPARK-3019, by Sandy Ryza and Mridul Muralidharan

        Attachments

        1. SPARK-19659-design-v1.pdf
          63 kB
          Jin Xing
        2. SPARK-19659-design-v2.pdf
          60 kB
          Jin Xing

          Issue Links

            Activity

              People

              • Assignee:
                jinxing6042@126.com Jin Xing
                Reporter:
                jinxing6042@126.com Jin Xing
              • Votes:
                0 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: