[HDFS-6981] Fix DN upgrade with layout version change

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Arpit Agarwal
• Reporter: James Thomas
• Created: 01/Sep/14 21:40
• Updated: 12/May/16 18:18
• Resolved: 09/Sep/14 04:22

Description

Post HDFS-6800, we can encounter the following scenario:

1. We start with DN software version -55 and initiate a rolling upgrade to version -56
2. We delete some blocks, and they are moved to trash
3. We roll back to DN software version -55 using the -rollback flag – since we are running the old code (prior to this patch), we will restore the previous directory but will not delete the trash
4. We append to some of the blocks that were deleted in step 2
5. We then restart a DN that contains blocks that were appended to – since the trash still exists, it will be restored at this point, the appended-to blocks will be overwritten, and we will lose the appended data

So I think we need to avoid writing anything to the trash directory if we have a previous directory.

Thanks to james.thomas for reporting this.

Issue Links

• is required by: HDFS-6581 Write to single replica in memory
• requires: HDFS-6482 Use block ID-based block layout on datanodes
• requires: HDFS-6800 Support Datanode layout changes with rolling upgrade