Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19937

Collect metrics of block sizes when shuffle.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.3.0
    • Component/s: Shuffle
    • Labels:
      None

      Description

      This improvement is related to SPARK-19659 and also documented there. Metrics of blocks sizes(when shuffle) should be collected for later analysis. This is helpful for analysis when skew situations or OOM happens(though maxBytesInFlight is set).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jinxing6042@126.com jin xing
                Reporter:
                jinxing6042@126.com jin xing
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: