[HADOOP-15132] Über-jira: WASB client phase III: roll-up for Hadoop 3.2

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: fs/azure
• Target Version/s: 3.5.0
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 19/Dec/17 16:27
• Updated: 04/Jan/24 12:23

Description

Everything for the WASB connector for Hadoop 3.1

Issue Links

• depends upon: HADOOP-14943 Add common getFileBlockLocations() emulation for object stores, including S3A
• supercedes: HADOOP-14552 Über-jira: WASB client phase II: performance and testing

Sub-Tasks

• Block compaction for WASB (Block Blobs Instead of Page Blobs): Open
• Hadoop-Azure Update WASB URI format to support SAS token in it.: Open
• WASB's file delete still throwing Blob not found exception: Open
• Implement isFileClosed in WASB to make Flume happy: Open
• Improve efficiency of WASB over page blobs: Open
• Support WASB connections through an HTTP proxy server.: Open
• WASB: Expose WASB status metrics as counters in Hadoop: Open
• wasb to support getStorageStatistics: Open
• General improvements to Azure test cases: Open
• WASB to support high-performance commit protocol: Resolved
• azureFs::getFileStatus doesn't check the file system scheme and thus could throw a misleading exception.: Patch Available
• ITestAzureConcurrentOutOfBandIo failing: The MD5 value specified in the request did not match with the MD5 value calculated by the server: Open
• AzureNativeFileSystemStore.getDirectorySet() to use getTrimmedStrings to get directory listings: Open
• Azure TestClientThrottlingAnalyzer brittle: Open
• Wasb getFileBlockLocations() returns too many locations.: Resolved
• add AbstractContractRootDirectoryTest test for mkdir / ; wasb to implement the test: Patch Available
• NativeAzureFileSystem file rename is not atomic: Resolved
• WASB to implement copyFromLocalFile(): Resolved
• WASB to implement listFiles(Path f, boolean recursive) through flat list: Resolved
• Wasb input streams to implement CanUnbuffer: Resolved
• NativeAzureFileSystem.open() ignores blob container name: Open
• TestNativeAzureFileSystemOperationsMocked creating files in /tmp: Open
• Reduce memory copy in BlobOutputStreamInternal::dispatchWrite: Open
• NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString() values: Patch Available
• WASB exception handling to translate specific failures into specific exceptions: Open
• WASB test runs leak storage containers.: Open
• backport HADOOP-15086 rename fix to branch-2: Resolved
• provide non-zero default for the Azure rename & delete thread pool sizes: Open
• NativeAzureFileSystemBaseTest.testModifiedTimeForFolder modification tests fail on (overloaded?) client: Open
• wasb FS init hangs for minutes if the container name is wrong (e.g wasb://example.azure/): Open
• Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks: Resolved
• Make WASB listStatus messages consistent: Resolved
• WASB: Test failures: Resolved