Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3326

ReduceTask should not sleep for 200 ms while waiting for merge to finish

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.18.0
    • None
    • None
    • Reviewed
    • Changed fetchOutputs() so that LocalFSMerger and InMemFSMergeThread threads are spawned only once. The thread gets notified when something is ready for merge. The merge happens when thresholds are met.

    Description

      Currently the merge code in Reduce task does:

                  // Wait for the on-disk merge to complete
                  while (localFSMergeInProgress) {
                    Thread.sleep(200);
                  }
                  
                  //wait for an ongoing merge (if it is in flight) to complete
                  while (mergeInProgress) {
                    Thread.sleep(200);
                  }
      

      Attachments

        1. hadoop-3326_4.patch
          25 kB
          Mahadev Konar
        2. 3326_3.patch
          25 kB
          Sharad Agarwal
        3. 3326_2.patch
          22 kB
          Sharad Agarwal
        4. 3326_1.patch
          21 kB
          Sharad Agarwal

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sharadag Sharad Agarwal Assign to me
            omalley Owen O'Malley
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment