[HADOOP-15763] Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: fs/azure
• Assignee: Da Zhou
• Reporter: Steve Loughran
• Created: 17/Sep/18 22:19
• Updated: 23/Jun/21 11:51
• Resolved: 23/Jun/21 11:51

Description

ABFS phase II: address issues which surface in the field; tune things which need tuning, add more tests where appropriate. Improve docs, especially troubleshooting. Classpaths. The usual.

Issue Links

• depends upon: HADOOP-15407 Support Windows Azure Storage - Blob file system in Hadoop
• is superceded by: HADOOP-17736 Über-JIRA: abfs phase III: Hadoop 3.3.2 features & fixes
• relates to: HADOOP-16353 Backport ABFS changes from trunk to branch-3.2

Sub-Tasks

• ABFS checkException to map 403 to AccessDeniedException: Resolved
• intermittent failure of TestAbfsClient.validateUserAgent: Resolved
• ABFS: Better exception handling of DNS errors: Resolved
• ABFS: validate packing, transient classpath, hadoop fs CLI: Resolved
• Document ABFS: Resolved
• typo in AzureBlobFileSystem.getIsNamespaceEnabeld: Resolved
• Make HTTPS the default protocol for ABFS: Resolved
• ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console: Resolved
• ABFS: Stop requiring client ID and tenant ID for MSI: Resolved
• ABFS: Enable some tests for namespace not enabled account using OAuth: Resolved
• ABFS: change createRemoteFileSystemDuringInitialization default to true: Resolved
• ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries.: Resolved
• ABFS: Throw IllegalArgumentException when Directory/File name ends with a period(.): Resolved
• Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore: Resolved
• ABFS: make retry policy configurable: Resolved
• ABFS: For HNS account, avoid unnecessary get call when doing Rename: Resolved
• ABFS: replace $superuser with local user: Resolved
• ABFS: Enable owner and group conversion for MSI and login user using OAuth: Resolved
• ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2: Resolved
• ABFS: add try catch for UGI failure when initializing ABFS: Resolved
• ABFS: getNamespaceEnabled can fail blocking user access thru ACLs: Resolved
• ABFS: reduce list page size to to 500: Resolved
• ABFS: remove timeout check for DELETE and RENAME: Resolved
• ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException: Resolved
• ABFS: Bug fix for tolerateOobAppends configuration: Resolved
• UserAgent string for ABFS: Resolved
• ABFS: Better exception handling of DNS errors followup: Resolved
• ABFS: Fix Date format parser: Resolved
• ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store: Resolved
• NativeAzureFileSystem does not support setXAttr: Resolved
• Failure of ABFS test ITestAbfsIdentityTransformer: Resolved
• Wasb tests to downgrade to skip when test a/c is namespace enabled: Resolved
• ABFS: Should only transform username to short name: Resolved
• hadoop fs mkdir / of nonexistent abfs container raises NPE: Resolved
• NPE in ABFS Client Credential Auth: Resolved
• NPE in setup/teardown of ITestAbfsDelegationTokens: Resolved
• ABFS: Bug fix for getPathProperties: Resolved
• Update abfs storage back-end with "close" flag when application is done writing to a file: Resolved
• ABFS: add bufferpool to AbfsOutputStream: Resolved
• ABFS: add FSMainOperationsBaseTest: Resolved
• ABFS: add listFileStatus with StartFrom: Resolved
• ABFS: transform full UPN for named user in AclStatus: Resolved
• ABFS driver continues to retry on IOException responses from REST operations: Resolved
• ABFS: Override access() to no-op for now: Resolved
• ABFS: port Azure doc to 3.2 branch: Resolved
• ABFS default blocksize change(256MB from 512MB): Resolved
• ABFS: Implement FileSystem.access() method: Resolved
• ABFS FileStatus.getModificationTime returns localized time instead of UTC: Resolved
• AzureADAuthenticator cannot authenticate in china: Resolved
• Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property: Resolved
• ABFS: Add missing query parameter for getPathStatus: Resolved
• ABFS: Config to enable/disable flush operation: Resolved
• ABFS: fileSystemExists() should not call container level apis: Resolved
• Make AAD endpoint configurable on all Auth flows: Resolved
• Track Azure Blob File System client-perceived latency: Resolved
• Backport HADOOP-16548 - "ABFS: Config to enable/disable flush operation" to branch-2: Resolved
• Backport HADOOP-16578 - "ABFS: fileSystemExists() should not call container level apis" to Branch-2: Resolved
• Backport HADOOP-16587 - "Make AAD endpoint configurable on all Auth flows" to branch-2: Resolved
• ABFS: add missing docs for configuration: Resolved
• ABFS: Make RetryCount in ExponentialRetryPolicy Configurable: Resolved
• Remove unnecessary ABFS toString() invocations: Resolved
• ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account: Resolved
• ABFS: Enhance driver debug logs: Resolved
• ABFS: Add additional tests for CustomTokenProvider: Resolved
• ITestClientUrlScheme fails for accounts which don't support HTTP: Resolved
• NPE in UGI.getCurrentUser in ITestAbfsIdentityTransformer setup: Resolved
• Backport HADOOP-16612 - "Track Azure Blob File System client-perceived latency" to branch-2: Resolved
• Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to branch-2: Resolved
• Backport HADOOP-16548 - ABFS: Config to enable/disable flush operation issue to branch-3.2: Resolved
• ABFS: Backport HADOOP-16660 ABFS: Make RetryCount in ExponentialRetryPolicy Configurable to Branch-2: Resolved
• Backport HADOOP-16612 "Track Azure Blob File System client-perceived latency" to branch-2: Resolved
• Improve wasb and abfs resilience on double close() calls: Resolved
• ABFS: Combine append+flush calls for blockblob & appendblob: Resolved
• ITestAzureBlobFileSystemCheckAccess failing: Resolved
• ABFS: update abfs.md to include config keys for identity transformation: Resolved
• ITestAbfsClient.testContinuationTokenHavingEqualSign failing: Resolved
• ABFS: Send error back to client for Read Ahead request failure: Resolved
• ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure: Resolved
• ABFS: Optimize HttpRequest retry triggers: Resolved
• ABFS: Add unbuffer support to AbfsInputStream: Resolved
• ABFS: Change in expiry calculation for MSI token provider: Resolved
• ABFS: Support for OAuth v2.0 endpoints: Resolved
• ABFS: Test failure ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance: Resolved
• ABFS: Make list page size configurable: Resolved
• ABFS: Change in User-Agent header: Resolved
• Backport HADOOP-16890- "ABFS: Change in expiry calculation for MSI token provider" & HADOOP-16825 "ITestAzureBlobFileSystemCheckAccess failing" to branch-2: Resolved
• ABFS: Support infinite lease dirs: Resolved
• ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics): Resolved
• ABFS: Upgrade Store REST API Version to 2019-12-12: Resolved
• ABFS: Avoid storage calls to check if the account is HNS enabled or not: Resolved
• ABFS: Improve the ABFS driver documentation: Resolved
• ABFS: Make PUT and POST operations idempotent: Resolved
• Support disabling buffered reads in ABFS positional reads: Resolved
• ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting: Resolved
• ABFS: Fix idempotency test failures when SharedKey is set as AuthType: Resolved
• Support for Appendblob in abfs driver: Resolved
• javadoc failing in the yetus report with the latest trunk: Resolved
• ABFS: Fix the parsing errors in ABFS Driver with creation Time (being returned in ListPath): Resolved
• ABFS: Long waits and unintended retries when multiple threads try to fetch token using ClientCreds: Resolved
• ABFS: GetAccessToken unrecoverable failures are being retried: Resolved
• ABFS: Fix For Idempotency code: Resolved
• ABFS: Tests ITestAbfsNetworkStatistics need to be config setting agnostic: Resolved
• ABFS: Test failure: testFailedRequestWhenCredentialsNotCorrect fails when run with SharedKey: Resolved
• ABFS: Test failure: Disable ITestAzureBlobFileSystemDelegationSAS tests: Resolved
• ABFS: Add debug log for rename failures: Resolved
• ABFS: configure output stream thread pool: Resolved
• ABFS: Enable checkaccess API: Resolved
• ABFS: Run the integration tests with various combinations of configurations and publish consolidated results: Resolved
• ABFS: Support for conditional overwrite: Resolved
• ABFS: Support for Client Correlation ID: Resolved
• ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account: Resolved
• ABFS: Allow Random Reads to be of Buffer Size: Resolved
• ABFS: read-ahead error reporting breaks buffer management: Resolved
• ABFS: Logs should redact SAS signature: Resolved
• Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-3.2: Resolved
• Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" and HADOOP-16785. "Improve wasb and abfs resilience on double close() calls. followup to abfs close() fix." to branch-2.10: Resolved
• ABFS: read/cache footer with fs.azure.footer.read.request.size: Resolved
• ABFS: testRenameFileOverExistingFile Fails after Contract test update: Resolved
• ABFS: SAS Test updates for version and permission update: Resolved
• ABFS: Piggyback flush on Append calls for short writes: Resolved
• ABFS: Delete Idempotency handling can lead to NPE: Resolved
• ABFS: Release Elastic ByteBuffer pool memory at outputStream close: Resolved
• ABFS: Set default ListMaxResults to max server limit: Resolved
• Optimise abfs incremental listings: Resolved
• ABFS: Fix auth failures getting counted as throttling errors in ABFS throttling interceptor: Resolved
• ABFS: Fix boundary conditions in InputStream seek and skip: Resolved
• ABFS: ITestAzureBlobFileSystemCheckAccess test failure if test doesn't have oauth keys: Resolved
• ABFS: Suport for customer provided encryption key: Resolved
• Correct abfs test assertion reversed in HADOOP-13327: Resolved
• ABFS: Toggle Store Mkdirs request overwrite parameter: Resolved
• ABFS: Disable throttling update for auth failures: Resolved