Hadoop Common
  1. Hadoop Common
  2. HADOOP-3427

In ReduceTask::fetchOutputs, wait for result can be improved slightly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed shuffle scheduler policy to wait for notifications from shuffle threads before scheduling more.

      Description

      The getCopyResult call in ReduceTask::fetchOutputs waits for 2 seconds if the results list is empty. This can be improved to wait only when sufficient number of fetches (above a certain threshold) have been scheduled. The threshold should ensure that all the fetcher threads would be busy enough, and in this case, the call to getCopyResult would return only when a fetcher thread wakes it up.

      1. 3427.patch
        17 kB
        Devaraj Das
      2. 3427.patch
        17 kB
        Devaraj Das
      3. 3427.patch
        17 kB
        Devaraj Das
      4. 3427.3.patch
        17 kB
        Devaraj Das

        Activity

          People

          • Assignee:
            Devaraj Das
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development