[HADOOP-18072] Über-JIRA: abfs phase III: Hadoop 3.3.x features & fixes

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.2
• Component/s: fs/azure
• Assignee: Anmol Asrani
• Reporter: Steve Loughran
• Created: 07/Jan/22 14:09
• Updated: 14/Dec/22 10:14

Description

ABFS related changes for hadoop 3.3.3

Issue Links

• is a parent of: HADOOP-18516 [ABFS]: Support fixed SAS token config in addition to SAS Token Provider class
• is blocked by: HADOOP-17474 Optimise abfs incremental listings

Sub-Tasks

• ABFS: rename ListResultSchemaTest to TestListResultSchema so maven tests run it: Open
• ABFS: Remove the workaround used for Http PATCH: Open
• ABFS: Improve test scripts: Patch Available
• ABFS: Enable optional store connectivity over azure specific protocol for data egress: Open
• ITestAbfsInputStreamStatistics#testReadAheadCounters timing out always: Reopened
• ABFS: Modifications to TracingContext: Open
• ABFS: Stabilize openFile withStatus: Resolved
• Add support for Filesystem#getFileChecksum in ABFS driver: Open
• ABFS: Refactor HTTP request handling code: Open
• ABFS tests to include terasort: Resolved
• ABFS: improve random read performance: Open
• Abfs directory delete times out on large directory tree w/ Oauth: OperationTimedOut: Open
• ABFS: Increase timeout of ITestAbfsReadWriteAndSeek: Patch Available
• ABFS VersionedFileStatus to declare that it isEncrypted(): Patch Available
• ABFS: Testcase failure ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics: Patch Available
• ABFS: ITestAzureBlobFileSystemLease test failure for Appendblob HNS OAuth: Open
• ABFS: Send ReadAhead feature utilization metrics to server: Open
• ABFS: Refactor read flow to include ReadRequestParameter: Open
• ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side: Open
• abfs can't access storage account without password: Open
• AbstractAbfsIntegrationTest.createFilesystemForSASTests() leaks FS instances: Open
• ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters: Resolved
• ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting: Open
• ABFS:Support for secondary accounts on ABFS Driver for SharedKey Auth: Open
• ABFS: Introduce Lease Operations with Append to provide single writer semantics: Open
• ABFS: Use Unique File Paths in Tests: Resolved
• Pass exception to the caller in AbfsClient#appendSASTokenToQuery: Open
• ABFS to support a Delegation Token provider which marshalls current Oauth secrets: Open
• Review/implement ABFS support for the extra fs ops which some apps (HBase) expects: Open
• ABFS: Implementation for getContentSummary: Open
• ABFS: test failure testLastModifiedTime(): Reopened
• ABFS IsNamespaceEnabled shouldn't use/log exceptions as default execution path: Open
• ABFS test setup failing "The specified filesystem already exists": Open
• Add ABFS support for Async Scatter/Gather IO: Open
• ABFS: Fix for OutofMemoryException from AbfsOutputStream: Open
• ABFS: Have all external dependent module execution tracked with DurationInfo: Open
• ABFS: transient failure of TestAzureBlobFileSystemFileStatus.testLastModifiedTime: Open
• ABFS AbfsDelegationTokenManager to generate canonicalServiceName if DT plugin doesn't: Open
• WASB: Fix Compiler Warnings: Open
• ABFS: Enable config controlled ETag check for Rename idempotency: Open
• abfs can't access storage account if soft delete is enabled: Open
• ABFS: Support for Encryption Context: Open
• ABFS: Test testAbfsStreamOps timing out: Open
• ABFS: Set driver global timeout for ITestAzureBlobFileSystemBasics: Resolved
• ABFS: Fork AbfsHttpOperation to add alternate connection: Open
• transient failure of ITestAbfsListStatusRemoteIterator.testWithAbfsIteratorDisabledWithoutHasNext: Open
• ABFS: ITestAbfsStreamStatistics TestAbfsStreamOps fail with append blob on HNS account: Open
• ABFS Adding an option to Always use Read Ahead, even for non sequential reads: Patch Available
• ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException: Reopened
• ABFS: Test with 100MB buffer size in ITestAbfsReadWriteAndSeek times out: Open
• ABFS: delete call sets Socket timeout lesser than query timeout leading to failures: Open
• ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator: Resolved
• ABFS to log @ debug each config option *And where it comes from*: Open
• ABFS: Enable config control for default connection timeout: Open
• ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException: Resolved
• ABFS: Skip testEtagConsistencyAcrossRename for Non-HNS accounts: Open
• ABFS: Toggle Store Mkdirs request overwrite parameter with default value: Open
• FileNotFoundException in abfs mkdirs() call: Open
• ABFS: Add testfilePath while verifying test contents are read correctly: Resolved
• ABFS: Lease operations: Open
• ABFS: Add changes for expect hundred continue header with append requests: Resolved
• ABFS: Add support for cache handling when filesystem instance has clientCorrelationID enabled: Open
• ABFS: Fix FileSystemAlreadyExists error for AzureAD errors: Open
• ABFS: Add correlated metric support for ABFS operations: Open
• abfs testReadAndWriteWithDifferentBufferSizesAndSeek failure: Open
• ABFS: Support for account level throttling: Resolved
• Add reason in in x-ms-client-request-id on a retry API call.: Resolved
• ABFS: Customize and optimize timeouts made based on each separate request: Open
• ABFS: Enabling Client-side Backoff only for new requests: Open
• x-ms-client-request-id to have some way that identifies retry of an API.: Open
• ABFS: Support for Pagination in Recursive Directory Delete: Open
• ABFS: eTag Check to mitigate Rename Failures: Resolved