Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3517

The last InMemory merge may be missed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.18.0
    • 0.18.0
    • None
    • None
    • Reviewed

    Description

      This is post HADOOP-3366. The inmem merge thread has the loop:

              while (!exitInMemMerge) {
                  ramManager.waitForDataToMerge();
                  doInMemMerge();
                }
      

      The fetchOutputs, at the end of copying everything, does the following:

              exitInMemMerge = true; 
              ramManager.close();
      

      Now if the merge thread is doing a merge (inside the doInMemMerge method) when the exitInMemMerge is set to true, the loop will break and the last merge of the files that got shuffled recently will be skipped. ramManager.close(), that internally does a final notify to the merge thread also won't have any effect in this case.

      Attachments

        1. HADOOP-3517_2_20080611.patch
          15 kB
          Arun Murthy
        2. HADOOP-3517_2_20080611.patch
          92 kB
          Arun Murthy
        3. HADOOP-3517_2_20080611.patch
          18 kB
          Arun Murthy
        4. HADOOP-3517_1_20080610.patch
          14 kB
          Arun Murthy

        Issue Links

          Activity

            People

              acmurthy Arun Murthy
              ddas Devaraj Das
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: