Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27874

ShuffleBlockFetchIterator can log information about per-block sizes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Shuffle, Spark Core
    • Labels:
      None

      Description

      It can be beneficial for monitoring and debugging to know the size of the largest shuffle block (or some distribution of block sizes). The ShuffleBlockFetcherIterator currently logs the number of local & remote blocks and their total size (see SPARK-26697), but does not log any information about per-block sizes.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bjornjon Bjorn Jonsson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: