Hadoop Common
  1. Hadoop Common
  2. HADOOP-3326

ReduceTask should not sleep for 200 ms while waiting for merge to finish

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed fetchOutputs() so that LocalFSMerger and InMemFSMergeThread threads are spawned only once. The thread gets notified when something is ready for merge. The merge happens when thresholds are met.

      Description

      Currently the merge code in Reduce task does:

                  // Wait for the on-disk merge to complete
                  while (localFSMergeInProgress) {
                    Thread.sleep(200);
                  }
                  
                  //wait for an ongoing merge (if it is in flight) to complete
                  while (mergeInProgress) {
                    Thread.sleep(200);
                  }
      
      1. 3326_1.patch
        21 kB
        Sharad Agarwal
      2. 3326_2.patch
        22 kB
        Sharad Agarwal
      3. 3326_3.patch
        25 kB
        Sharad Agarwal
      4. hadoop-3326_4.patch
        25 kB
        Mahadev konar

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Sharad Agarwal
            Reporter:
            Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development