[HADOOP-14552] Über-jira: WASB client phase II: performance and testing

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 2.9.0, 3.0.0
• Component/s: fs/azure
• Assignee: Thomas Marqardt
• Reporter: Steve Loughran
• Created: 20/Jun/17 10:26
• Updated: 19/Dec/17 17:11
• Resolved: 19/Dec/17 17:09

Description

Uber-JIRA for wasb:// phase II: the things we want for Hadoop 2.9.

I think the focus is on performance, but looking at the tests there are some things we need there first: parallel test execution, move over to IT tests, other tuning.

All patches must come with declarations of which azure endpoint they were tested against.

Issue Links

• depends upon: HADOOP-14547 [WASB] the configured retry policy is not used for all storage operations.
• depends upon: HADOOP-14518 Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
• depends upon: HADOOP-14461 Azure: handle failure gracefully in case of missing account access key
• depends upon: HADOOP-14478 Optimize NativeAzureFsInputStream for positional reads
• depends upon: HADOOP-14581 Restrict setOwner to list of user when security is enabled in wasb
• depends upon: HADOOP-14598 Blacklist Http/HttpsFileSystem in FsUrlStreamHandlerFactory
• depends upon: HADOOP-14443 Azure: Support retry and client side failover for authorization, SASKey and delegation token generation
• depends upon: HADOOP-14520 WASB: Block compaction for Azure Block Blobs
• depends upon: HADOOP-14535 wasb: implement high-performance random access and seek of block blobs
• depends upon: HADOOP-14536 Update azure-storage sdk to version 5.3.0
• is superceded by: HADOOP-15132 Über-jira: WASB client phase III: roll-up for Hadoop 3.2
• relates to: HADOOP-14478 Optimize NativeAzureFsInputStream for positional reads
• relates to: HADOOP-14680 Azure: IndexOutOfBoundsException in BlockBlobInputStream
• relates to: HADOOP-15042 Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0

Sub-Tasks

• Customize User-Agent header sent in HTTP/HTTPS requests by WASB.: Resolved
• Upgrade azure-storage sdk version >5.4.0: Resolved
• Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients: Closed
• Remove service loader config file for wasb fs: Resolved
• Add (parallelized) integration tests to hadoop-azure: Resolved
• TestFileSystemOperationExceptionMessage to not rerun all of NativeAzureFileSystemBaseTest: Resolved
• document how to run wasb tests in azure docs site/testing.md: Resolved
• [WASB] the configured retry policy is not used for all storage operations.: Resolved
• regression: Azure tests which capture logs failing with move to SLF4J: Resolved
• wasb throws an exception if you try to create a file and there's no parent directory: Closed
• Azure: Concurrent I/O does not work when secure.mode is enabled: Resolved
• Azure: Reuse ObjectMapper: Resolved
• NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code: Resolved
• error message in AzureNativeFileSystemStore.connectUsingAnonymousCredentials has too many spaces: Resolved
• wasb: add support for caching Authorization and SASKeys: Resolved
• wasb: improve throughput by 34% when account limit exceeded: Resolved
• TestWasbRemoteCallHelper failing: Resolved
• Honoring sticky bit during Deletion when authorization is enabled in WASB: Resolved
• WASB: delete recursive should not fail if a file is deleted: Resolved
• PageBlobOutputStream to declare hflush/hsync StreamCapabilities for: Resolved
• Add support for using container saskeys for all accesses: Resolved
• Azure wasb: getFileStatus not making any auth checks: Resolved
• Restrict Access to setPermission operation when authorization is enabled in WASB: Resolved
• Azure: POSIX permissions are taking effect in access() method even when authorization is enabled: Resolved
• Sticky bit implementation for rename() operation in Azure WASB: Resolved
• ITestNativeAzureFileSystemLive.testLeaseAsDistributedLock assert failure in parallel run: Resolved