[HADOOP-12884] A failed atomic folder rename operation can never recovery for the destination file is deleted in Wasb filesystem

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Shaohui Liu
• Created: 03/Mar/16 14:05
• Updated: 04/Mar/16 02:11
• Resolved: 04/Mar/16 02:11

Description

When using Wsab file system, we found that a failed atomic folder rename operation can never recovery for the destination file deleted in Wasb filesystem.

ls: Attempting to complete rename of file hbase/azurtst-xiaomi/data/default/YCSBTest/.tabledesc during folder rename redo, and file was not found in source or destination.

The reason is the the file is renamed to the destination file before the crash, and the destination file is deleted by another process after crash. So the recovery is blocked during finishing the rename operation of this file when found the source and destination files all don't exist.

See: NativeAzureFileSystem.java #finishSingleFileRename

Another serious problem is that the recovery of atomic rename operation may delete new created file which is same name as the source file, because the file system don't check if there are rename operation need be redo.

Suggestions are welcomed~

Issue Links

• is duplicated by: HADOOP-12634 Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy
• is duplicated by: HADOOP-12780 During WASB atomic rename handle crash when one directory has been renamed but not file under it.