Hadoop Common
  1. Hadoop Common
  2. HADOOP-3427

In ReduceTask::fetchOutputs, wait for result can be improved slightly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed shuffle scheduler policy to wait for notifications from shuffle threads before scheduling more.

      Description

      The getCopyResult call in ReduceTask::fetchOutputs waits for 2 seconds if the results list is empty. This can be improved to wait only when sufficient number of fetches (above a certain threshold) have been scheduled. The threshold should ensure that all the fetcher threads would be busy enough, and in this case, the call to getCopyResult would return only when a fetcher thread wakes it up.

      1. 3427.3.patch
        17 kB
        Devaraj Das
      2. 3427.patch
        17 kB
        Devaraj Das
      3. 3427.patch
        17 kB
        Devaraj Das
      4. 3427.patch
        17 kB
        Devaraj Das

        Activity

        Devaraj Das created issue -
        Devaraj Das made changes -
        Field Original Value New Value
        Attachment 3427.patch [ 12382537 ]
        Devaraj Das made changes -
        Attachment 3427.patch [ 12382546 ]
        Devaraj Das made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Devaraj Das made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Devaraj Das made changes -
        Attachment 3427.patch [ 12382819 ]
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Devaraj Das made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Devaraj Das made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Devaraj Das made changes -
        Attachment 3427.3.patch [ 12383448 ]
        Devaraj Das made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags [Reviewed]
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Release Note Improves the shuffle scheduler. It now waits for notifications from shuffle threads when it has scheduled enough, before scheduling more.
        Robert Chansler made changes -
        Release Note Improves the shuffle scheduler. It now waits for notifications from shuffle threads when it has scheduled enough, before scheduling more. Changed shuffle scheduler policy to wait for notifications from shuffle threads before scheduling more.
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]

          People

          • Assignee:
            Devaraj Das
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development