Hadoop Common
  1. Hadoop Common
  2. HADOOP-3326

ReduceTask should not sleep for 200 ms while waiting for merge to finish

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed fetchOutputs() so that LocalFSMerger and InMemFSMergeThread threads are spawned only once. The thread gets notified when something is ready for merge. The merge happens when thresholds are met.

      Description

      Currently the merge code in Reduce task does:

                  // Wait for the on-disk merge to complete
                  while (localFSMergeInProgress) {
                    Thread.sleep(200);
                  }
                  
                  //wait for an ongoing merge (if it is in flight) to complete
                  while (mergeInProgress) {
                    Thread.sleep(200);
                  }
      
      1. 3326_1.patch
        21 kB
        Sharad Agarwal
      2. 3326_2.patch
        22 kB
        Sharad Agarwal
      3. 3326_3.patch
        25 kB
        Sharad Agarwal
      4. hadoop-3326_4.patch
        25 kB
        Mahadev konar

        Activity

        Owen O'Malley created issue -
        Devaraj Das made changes -
        Field Original Value New Value
        Assignee Sharad Agarwal [ sharadag ]
        Sharad Agarwal made changes -
        Attachment 3326_1.patch [ 12382454 ]
        Sharad Agarwal made changes -
        Attachment 3326_2.patch [ 12382560 ]
        Sharad Agarwal made changes -
        Hadoop Flags [Reviewed]
        Status Open [ 1 ] Patch Available [ 10002 ]
        Sharad Agarwal made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Sharad Agarwal made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Devaraj Das made changes -
        Fix Version/s 0.18.0 [ 12312972 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Mahadev konar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags [Reviewed]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Sharad Agarwal made changes -
        Attachment 3326_3.patch [ 12383464 ]
        Sharad Agarwal made changes -
        Release Note LocalFSMerger and InMemFSMergeThread threads are spawned only once in fetchOutputs() method. The thread gets notified when something is ready for merge. The merge happens when thresholds are met.
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Attachment hadoop-3326_4.patch [ 12383496 ]
        Arun C Murthy made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Devaraj Das made changes -
        Hadoop Flags [Reviewed]
        Robert Chansler made changes -
        Release Note LocalFSMerger and InMemFSMergeThread threads are spawned only once in fetchOutputs() method. The thread gets notified when something is ready for merge. The merge happens when thresholds are met. Changed fetchOutputs() so that LocalFSMerger and InMemFSMergeThread threads are spawned only once. The thread gets notified when something is ready for merge. The merge happens when thresholds are met.
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]

          People

          • Assignee:
            Sharad Agarwal
            Reporter:
            Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development