Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5873

Shuffle bandwidth computation includes time spent waiting for maps

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its definition however is confusing because it captures the time where there is no copying because there is a pause between when new wave of map outputs is available.
      current bw is definded as (bytes copied so far) / (total time in the copy phase so far)
      It would be more useful
      1) to measure bandwidth of a single copy call.
      2) display aggregated bw as long as there is at least one fetcher is in the copy call.

      1. MAPREDUCE-5873.v1.patch
        13 kB
        Siqi Li
      2. MAPREDUCE-5873.v2.patch
        15 kB
        Siqi Li
      3. MAPREDUCE-5873.v3.patch
        15 kB
        Siqi Li
      4. MAPREDUCE-5873.v4.patch
        18 kB
        Siqi Li
      5. MAPREDUCE-5873.v5.patch
        18 kB
        Siqi Li
      6. MAPREDUCE-5873.v6.patch
        18 kB
        Siqi Li
      7. MAPREDUCE-5873.v9.patch
        18 kB
        Siqi Li

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Jason Lowe made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.6.0 [ 12327180 ]
        Resolution Fixed [ 1 ]
        Jason Lowe made changes -
        Summary Measure bw of a single copy call and display the correct aggregated bw Shuffle bandwidth computation includes time spent waiting for maps
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v7.patch [ 12674604 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v8.patch [ 12674612 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v9.patch [ 12674635 ]
        Siqi Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v8.patch [ 12674612 ]
        Siqi Li made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siqi Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v7.patch [ 12674604 ]
        Siqi Li made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siqi Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v6.patch [ 12673678 ]
        Siqi Li made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siqi Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v5.patch [ 12673483 ]
        Siqi Li made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siqi Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v4.patch [ 12672869 ]
        Siqi Li made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siqi Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siqi Li made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v3.patch [ 12645282 ]
        Siqi Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v2.patch [ 12643617 ]
        Siqi Li made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siqi Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 2.3.0 [ 12325257 ]
        Assignee Siqi Li [ l201514 ]
        Siqi Li made changes -
        Attachment MAPREDUCE-5873.v1.patch [ 12642893 ]
        Siqi Li made changes -
        Description 1) to measure bandwidth of a single copy call.
        2) display aggregated bw as long as there is at least one fetcher is in the copy call.
        Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its definition however is confusing because it captures the time where there is no copying because there is a pause between when new wave of map outputs is available.
        current bw is definded as (bytes copied so far) / (total time in the copy phase so far)
        It would be more useful
        1) to measure bandwidth of a single copy call.
        2) display aggregated bw as long as there is at least one fetcher is in the copy call.
        Siqi Li made changes -
        Field Original Value New Value
        Description 1) to measure bandwidth of a single copy call.
        2) display aggregated bw as long as there is at least one fetcher is in the copy call.
        Siqi Li created issue -

          People

          • Assignee:
            Siqi Li
            Reporter:
            Siqi Li
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development