Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3326

ReduceTask should not sleep for 200 ms while waiting for merge to finish

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed fetchOutputs() so that LocalFSMerger and InMemFSMergeThread threads are spawned only once. The thread gets notified when something is ready for merge. The merge happens when thresholds are met.

      Description

      Currently the merge code in Reduce task does:

                  // Wait for the on-disk merge to complete
                  while (localFSMergeInProgress) {
                    Thread.sleep(200);
                  }
                  
                  //wait for an ongoing merge (if it is in flight) to complete
                  while (mergeInProgress) {
                    Thread.sleep(200);
                  }
      

        Attachments

        1. hadoop-3326_4.patch
          25 kB
          Mahadev konar
        2. 3326_3.patch
          25 kB
          Sharad Agarwal
        3. 3326_2.patch
          22 kB
          Sharad Agarwal
        4. 3326_1.patch
          21 kB
          Sharad Agarwal

          Activity

            People

            • Assignee:
              sharadag Sharad Agarwal
              Reporter:
              owen.omalley Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: