[HDFS-4739] NN can miscalculate the number of extra edit log segments to retain

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.4-alpha
• Fix Version/s: 2.1.0-beta
• Component/s: namenode
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Aaron Myers
• Reporter: Aaron Myers
• Created: 24/Apr/13 17:47
• Updated: 27/Aug/13 22:08
• Resolved: 24/Apr/13 23:33

Description

The code in NNStorageRetentionManager#purgeOldStorage is intended to place a cap on the number of extra edit log segments retained beyond what is strictly required to replay the FS history since the last fsimage. In fact this code currently places a limit on the total number of extra edit log segments. If the number of required segments is greater than the configured cap, there will be no data loss, but an ugly error will be thrown and the NN will fail to start.

The fix is simple, and in the meantime a work-around is just to raise the value of dfs.namenode.max.extra.edits.segments.retained and start the NN.