[HDFS-5590] Block ID and generation stamp may be reused when persistBlocks is set to false

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Hadoop Flags: Reviewed
• Assignee: Jing Zhao Assign to me
• Reporter: Jing Zhao
• Created: 02/Dec/13 20:04
• Updated: 24/Feb/14 20:58
• Resolved: 06/Dec/13 02:18

Description

In a cluster with non-HA setup and dfs.persist.blocks set to false, we may have data loss in the following case:

1. client creates file1 and requests a block from NN and get blk_id1_gs1
2. client writes blk_id1_gs1 to DN
3. NN is restarted and because persistBlocks is false, blk_id1_gs1 may not be persisted in disk
4. another client creates file2 and NN will allocate a new block using the same block id blk_id1_gs1 since block ID and generation stamp are both increased sequentially.

Now we may have two versions (file1 and file2) of the blk_id1_gs1 (same id, same gs) in the system. It will case data loss.