[HADOOP-158] dfs should allocate a random blockid range to a file, then assign ids sequentially to blocks in the file

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.1.0
• Assignee: Konstantin Shvachko
• Reporter: Doug Cutting
• Created: 22/Apr/06 00:06
• Updated: 08/Jul/09 16:41
• Resolved: 11/Oct/07 19:19

Description

A random number generator is used to allocate block ids in dfs. Sometimes a block id is allocated that is already used in the filesystem, which causes filesystem corruption.

A short-term fix for this is to simply check when allocating block ids whether any file is already using the newly allocated id, and, if it is, generate another one. There can still be collisions in some rare conditions, but these are harder to fix and will wait, since this simple fix will handle the vast majority of collisions.

Issue Links

• duplicates: HADOOP-146 potential conflict in block id's, leading to data corruption
• is duplicated by: HADOOP-1497 Possibility of duplicate blockids if dead-datanodes come back up after corresponding files were deleted