Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33235 Push-based Shuffle Improvement Tasks
  3. SPARK-42203

JsonProtocol should skip logging of push-based shuffle read metrics when push-based shuffle is disabled

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Shuffle
    • None

    Description

      This is a followup to SPARK-36620:

      When push-based shuffle is disabled (the default), I think that we should skip the logging of the new push-based shuffle read metrics. Because these metrics are logged for every task, they will add significant additional size to Spark event logs. It would be great to avoid this cost in cases where it's not necessary.

      Attachments

        Issue Links

          is caused by

          Sub-task - The sub-task of the issue SPARK-36620 Client side related push-based shuffle metrics

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              joshrosen Josh Rosen
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: