[HADOOP-3517] The last InMemory merge may be missed

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.18.0
• Fix Version/s: 0.18.0
• Hadoop Flags: Reviewed
• Assignee: Arun Murthy
• Reporter: Devaraj Das
• Created: 09/Jun/08 04:03
• Updated: 15/Jan/24 19:37
• Resolved: 12/Jun/08 10:55

Description

This is post HADOOP-3366. The inmem merge thread has the loop:

        while (!exitInMemMerge) {
            ramManager.waitForDataToMerge();
            doInMemMerge();
          }

The fetchOutputs, at the end of copying everything, does the following:

        exitInMemMerge = true; 
        ramManager.close();

Now if the merge thread is doing a merge (inside the doInMemMerge method) when the exitInMemMerge is set to true, the loop will break and the last merge of the files that got shuffled recently will be skipped. ramManager.close(), that internally does a final notify to the merge thread also won't have any effect in this case.

Issue Links

• is related to: HADOOP-18717 Move CodecPool getCompressor/getDecompressor logs to DEBUG