Hadoop Common
  1. Hadoop Common
  2. HADOOP-3062

Need to capture the metrics for the network ios generate by dfs reads/writes and map/reduce shuffling and break them down by racks

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: metrics
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Introduced additional log records for data transfers.

      Description

      In order to better understand the relationship between hadoop performance and the network bandwidth, we need to know
      what the aggregated traffic data in a cluster and its breakdown by racks. With these data, we can determine whether the network
      bandwidth is the bottleneck when certain jobs are running on a cluster.

      1. 3062-5.patch
        22 kB
        Chris Douglas
      2. 3062-4.patch
        22 kB
        Chris Douglas
      3. 3062-3.patch
        21 kB
        Chris Douglas
      4. 3062-2.patch
        17 kB
        Chris Douglas
      5. 3062-1.patch
        17 kB
        Chris Douglas
      6. 3062-0.patch
        12 kB
        Chris Douglas

        Issue Links

          Activity

          Runping Qi created issue -
          Chris Douglas made changes -
          Field Original Value New Value
          Attachment 3062-0.patch [ 12387698 ]
          Chris Douglas made changes -
          Hadoop Flags [Incompatible change]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.19.0 [ 12313211 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Chris Douglas made changes -
          Attachment 3062-1.patch [ 12388314 ]
          Chris Douglas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Link This issue incorporates HADOOP-3658 [ HADOOP-3658 ]
          Chris Douglas made changes -
          Assignee Chris Douglas [ chris.douglas ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Description
          In order to better understand the relationship between hadoop performance and the network bandwidth, we need to know
          what the aggregated traffic data in a cluster and its breakdown by racks. With these data, we can determine whether the network
          bandwidth is the bottleneck when certain jobs are running on a cluster.
          In order to better understand the relationship between hadoop performance and the network bandwidth, we need to know
          what the aggregated traffic data in a cluster and its breakdown by racks. With these data, we can determine whether the network
          bandwidth is the bottleneck when certain jobs are running on a cluster.
          Chris Douglas made changes -
          Attachment 3062-2.patch [ 12388488 ]
          Chris Douglas made changes -
          Attachment 3062-3.patch [ 12388574 ]
          Chris Douglas made changes -
          Attachment 3062-4.patch [ 12388647 ]
          Chris Douglas made changes -
          Attachment 3062-5.patch [ 12388696 ]
          Chris Douglas made changes -
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Robert Chansler made changes -
          Release Note Introduced additional log records for data transfers.
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Chris Douglas
              Reporter:
              Runping Qi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development