Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6958

Shuffle audit logger should log size of shuffle transfer

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.3
    • Component/s: None
    • Labels:
      None

      Description

      The shuffle audit logger currently logs the job ID and reducer ID but nothing about the size of the requested transfer. It calculates this as part of the HTTP response headers, so it would be trivial to log the response size. This would be very valuable for debugging network traffic storms from the shuffle handler.

        Attachments

        1. MAPREDUCE-6958.001.patch
          2 kB
          Jason Lowe
        2. MAPREDUCE-6958.002.patch
          2 kB
          Jason Lowe
        3. MAPREDUCE-6958.003.patch
          2 kB
          Jason Lowe
        4. MAPREDUCE-6958-branch-2.002.patch
          2 kB
          Jason Lowe
        5. MAPREDUCE-6958-branch-2.8.002.patch
          6 kB
          Jason Lowe

          Activity

            People

            • Assignee:
              jlowe Jason Lowe
              Reporter:
              jlowe Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: