Hadoop Common
  1. Hadoop Common
  2. HADOOP-4749

reducer should output input data size when shuffling is done

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added a new counter REDUCE_INPUT_BYTES.

      Description

      Sometimes we see a single slow reducer because of the load balancing problem. This information will be very useful to understand how imbalanced the load is.

      Should be easy to fix I guess, since reducer should have all information needed at the end of the shuffling phase.

      1. 4749.patch
        2 kB
        He Yongqiang

        Issue Links

          Activity

          Zheng Shao created issue -
          Zheng Shao made changes -
          Field Original Value New Value
          Summary reducer should output input data size and record count when shuffling is done reducer should output input data size when shuffling is done
          He Yongqiang made changes -
          Attachment ReduceTask.patch [ 12395476 ]
          Attachment Task.patch [ 12395477 ]
          He Yongqiang made changes -
          Attachment Task.patch [ 12395477 ]
          He Yongqiang made changes -
          Attachment ReduceTask.patch [ 12395476 ]
          He Yongqiang made changes -
          Attachment 4749.patch [ 12395533 ]
          He Yongqiang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Added a new counter REDUCE_INPUT_BYTES in Task class for recording reducer's inputbytes
          Affects Version/s 0.19.0 [ 12313211 ]
          Zheng Shao made changes -
          Release Note Added a new counter REDUCE_INPUT_BYTES in Task class for recording reducer's inputbytes HADOOP-4749. Added a new counter REDUCE_INPUT_BYTES. (Yongqiang He via zshao)
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Fix Version/s 0.20.0 [ 12313438 ]
          Arun C Murthy made changes -
          Assignee he yongqiang [ he yongqiang ]
          He Yongqiang made changes -
          Comment [ hi, Arun
              Thank you.




          Best regards,
           
          Yongqiang He
          2008-12-12

          Email: heyongqiang@software.ict.ac.cn
          Tel: 86-10-62600966(O)
           
          Research Center for Grid and Service Computing,
          Institute of Computing Technology,
          Chinese Academy of Sciences
          P.O.Box 2704, 100080, Beijing, China



          发件人: Arun C Murthy (JIRA)
          发送时间: 2008-12-12 17:10:57
          收件人: heyongqiang@software.ict.ac.cn
          抄送:
          主题: [jira] Assigned: (HADOOP-4749) reducer should output input datasize when shuffling is done


               [ https://issues.apache.org/jira/browse/HADOOP-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

          Arun C Murthy reassigned HADOOP-4749:
          -------------------------------------

              Assignee: he yongqiang


          --
          This message is automatically generated by JIRA.
          -
          You can reply to this email to add a comment to the issue online.
          ]
          Zheng Shao made changes -
          Link This issue is related to HADOOP-4845 [ HADOOP-4845 ]
          Robert Chansler made changes -
          Release Note HADOOP-4749. Added a new counter REDUCE_INPUT_BYTES. (Yongqiang He via zshao) Added a new counter REDUCE_INPUT_BYTES.
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]

            People

            • Assignee:
              He Yongqiang
              Reporter:
              Zheng Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development