[HADOOP-12634] Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Gaurav Kanade
• Reporter: Gaurav Kanade
• Due: 15/Dec/15
• Created: 10/Dec/15 22:10
• Updated: 30/Aug/16 01:20
• Resolved: 06/Jan/16 20:10

Description

HADOOP-12334 changed mode of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries. This was via client side copy. However a process crash when the copy is partially done would result in a scenario where the source and destination blobs will have different contents and lazy rename pending operation will not handle this thus causing data loss. We need to fix the lazy rename pending operation to address this issue

Issue Links

• duplicates: HADOOP-12884 A failed atomic folder rename operation can never recovery for the destination file is deleted in Wasb filesystem