[HADOOP-16874] For AzureNativeFS, when BlockCompaction is enabled, FileSystem.create(path).close() would throw exception.

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.9.2, 3.2.1
• Component/s: fs/azure

Environment

macOS Mojave 10.14.6

 

• Assignee: Unassigned
• Reporter: Xiaolei Liu
• Created: 19/Feb/20 17:32
• Updated: 07/Mar/22 19:05

Description

For AzureNativeFS, when BlockCompaction is enabled, FileSystem.create(path).close() would throw blob not existed exception.

Block Compaction Setting: fs.azure.block.blob.with.compaction.dir
Exception is thrown from close(), this would happen when no write happened. When actually write any content in the file, same context close() won't trigger the exception.

When BlockCompaction is not enabled, this issue won't happen.

Call Stack:

org.apache.hadoop.fs.azure.AzureException: Source blob _$azuretmpfolder$/956457df-4a3e-4285-bc68-29f68b9b36c4test1911.log does not exist.
org.apache.hadoop.fs.azure.AzureException: Source blob _$azuretmpfolder$/956457df-4a3e-4285-bc68-29f68b9b36c4test1911.log does not exist.
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2648)
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2608)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsOutputStream.restoreKey(NativeAzureFileSystem.java:1199)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsOutputStream.close(NativeAzureFileSystem.java:1068)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)