Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3427

In ReduceTask::fetchOutputs, wait for result can be improved slightly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.18.0
    • None
    • None
    • Reviewed
    • Changed shuffle scheduler policy to wait for notifications from shuffle threads before scheduling more.

    Description

      The getCopyResult call in ReduceTask::fetchOutputs waits for 2 seconds if the results list is empty. This can be improved to wait only when sufficient number of fetches (above a certain threshold) have been scheduled. The threshold should ensure that all the fetcher threads would be busy enough, and in this case, the call to getCopyResult would return only when a fetcher thread wakes it up.

      Attachments

        1. 3427.3.patch
          17 kB
          Devaraj Das
        2. 3427.patch
          17 kB
          Devaraj Das
        3. 3427.patch
          17 kB
          Devaraj Das
        4. 3427.patch
          17 kB
          Devaraj Das

        Activity

          People

            ddas Devaraj Das
            ddas Devaraj Das
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: