[HDFS-3440] should more effectively limit stream memory consumption when reading corrupt edit logs

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.2-alpha
• Hadoop Flags: Reviewed
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 18/May/12 00:27
• Updated: 03/Mar/14 19:16
• Resolved: 18/May/12 05:45

Description

Currently, we do in.mark(100MB) before reading an opcode out of the edit log. However, this could result in us usin all of those 100 MB when reading bogus data, which is not what we want. It also could easily make some corrupt edit log files unreadable.

We should have a stream limiter interface, that causes a clean IOException when we're in this situation, and does not result in huge memory consumption.

Issue Links

• relates to: HDFS-3049 During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
• relates to: HDFS-5995 TestFSEditLogLoader#testValidateEditLogWithCorruptBody gets OutOfMemoryError and dumps heap.