[HADOOP-19092] ABFS phase 4: post Hadoop 3.4.0 features

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: fs/azure
• Assignee: Sneha Vijayarajan
• Reporter: Steve Loughran
• Created: 27/Feb/24 11:08
• Updated: 20/Aug/24 17:10

Description

Uber-JIRA for ABFS work so we can close HADOOP-18072 as done for 3.4.0

Assuming 3.4.1 is a rapid roll of packing, dependencies and critical fixes, this should target 3.4.2 and beyond

Issue Links

• depends upon: MAPREDUCE-7474 [ABFS] Improve commit resilience and performance in Manifest Committer
• is blocked by: HADOOP-17474 Optimise abfs incremental listings

Sub-Tasks

• [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout failures: Resolved
• ABFS: Add correlated metric support for ABFS operations: Resolved
• ABFS: Enable config control for default connection timeout: Resolved
• [ABFS] Reverting Back Support of setXAttr() and getXAttr() on root path: Resolved
• [ABFS]: Support fixed SAS token config in addition to Custom SASTokenProvider Implementation: Resolved
• ABFS: Fixing Test Script Bug and Some Known test Failures in ABFS Test Suite: Resolved
• [ABFS]: FooterReadBufferSize should not be greater than readBufferSize: Resolved
• WASB Driver Deprecation and eventual removal: Resolved
• Test ITestAbfsRestOperationException#testAuthFailException is broken.: Resolved
• [ABFS]Prevent ABFS initialization for non-hierarchical-namespace account if Customer-provided-key configs given.: Resolved
• [ABFS] Enhancing Client-Side Throttling Metrics Updation Logic: Resolved
• ABFS: Fixing logic to determine HNS nature of account to avoid extra getAcl() calls: Resolved
• Fix for Javac Error of Unchecked Cast: Open
• ABFS: Fix FileSystemAlreadyExists error for AzureAD errors: Open
• ABFS: Toggle Store Mkdirs request overwrite parameter with default value: Open
• ABFS: Modifications to TracingContext: Open
• ABFS: Add sendMs and recvMs for all AbfsHttpOperation calls: Open
• ABFS:Support for secondary accounts on ABFS Driver for SharedKey Auth: Open
• ITestAbfsInputStreamStatistics#testReadAheadCounters timing out always: Reopened
• ABFS: ITestAzureBlobFileSystemLease test failure for Appendblob HNS OAuth: Open
• FileNotFoundException in abfs mkdirs() call: Open
• Pass exception to the caller in AbfsClient#appendSASTokenToQuery: Open
• abfs testReadAndWriteWithDifferentBufferSizesAndSeek failure: Open
• [ABFS][Retry Policy] Using hadoop-common code to refractor Abfs Retry Policy Implementation: Open
• ABFS: Test testAbfsStreamOps timing out: Open
• ABFS: Add support for cache handling when filesystem instance has clientCorrelationID enabled: Open
• ABFS: delete call sets Socket timeout lesser than query timeout leading to failures: Open
• ABFS test setup failing "The specified filesystem already exists": Open
• ABFS: Fix for OutofMemoryException from AbfsOutputStream: Open
• ABFS: Improve test scripts: Patch Available
• abfs can't access storage account if soft delete is enabled: Open
• ABFS Adding an option to Always use Read Ahead, even for non sequential reads: Patch Available
• ABFS to support a Delegation Token provider which marshalls current Oauth secrets: Open
• transient failure of ITestAbfsListStatusRemoteIterator.testWithAbfsIteratorDisabledWithoutHasNext: Open
• Abfs directory delete times out on large directory tree w/ Oauth: OperationTimedOut: Open
• ABFS: Testcase failure ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics: Patch Available
• ABFS: Support for Pagination in Recursive Directory Delete: Resolved
• Add support for Filesystem#getFileChecksum in ABFS driver: Open
• ABFS: improve random read performance: Open
• ABFS VersionedFileStatus to declare that it isEncrypted(): Patch Available
• ABFS: Implementation for getContentSummary: Open
• ABFS: AbfsOutputStream doesnt close DataBlocks object.: Open
• AbstractAbfsIntegrationTest.createFilesystemForSASTests() leaks FS instances: Open
• WASB: Fix Compiler Warnings: Open
• ABFS: Introduce Lease Operations with Append to provide single writer semantics: Open
• ABFS: ITestAbfsStreamStatistics TestAbfsStreamOps fail with append blob on HNS account: Open
• ABFS: rename ListResultSchemaTest to TestListResultSchema so maven tests run it: Open
• ABFS: Refactor HTTP request handling code: Open
• Add ABFS support for Async Scatter/Gather IO: Open
• ABFS IsNamespaceEnabled shouldn't use/log exceptions as default execution path: Open
• ABFS: Lease operations: Open
• ABFS: Fork AbfsHttpOperation to add alternate connection: Open
• ABFS: Enable optional store connectivity over azure specific protocol for data egress: Open
• ABFS: Refactor read flow to include ReadRequestParameter: Open
• ABFS: test failure testLastModifiedTime(): Reopened
• ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side: Open
• ABFS to log @ debug each config option *And where it comes from*: Open
• ABFS AbfsDelegationTokenManager to generate canonicalServiceName if DT plugin doesn't: Open
• ABFS: transient failure of TestAzureBlobFileSystemFileStatus.testLastModifiedTime: Open
• ABFS: Send ReadAhead feature utilization metrics to server: Open
• ABFS: Have all external dependent module execution tracked with DurationInfo: Open
• ABFS: Customize and optimize timeouts made based on each separate request: Open
• ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException: Reopened
• Review/implement ABFS support for the extra fs ops which some apps (HBase) expects: Open
• ABFS: Remove the workaround used for Http PATCH: Open
• ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting: Open
• ABFS: Enabling Client-side Backoff only for new requests: Open
• ABFS contract-tests with Hadoop-Commons intermittently failing: Open
• ABFS: Remove commons IOUtils.close() from AbfsOutputStream: Open
• ABFS: Skip testEtagConsistencyAcrossRename for Non-HNS accounts: Open
• ABFS: Increase timeout of ITestAbfsReadWriteAndSeek: Patch Available
• ABFS: Test with 100MB buffer size in ITestAbfsReadWriteAndSeek times out: Open
• abfs can't access storage account without password: Open
• [ABFS] Support VectorIO in ABFS Input Stream: Open
• Improve rate limiting through ABFS in Manifest Committer: Open
• [ABFS]: ApacheHttpClient adaptation as network library: Resolved
• [ABFS] testListPathWithValueGreaterThanServerMaximum assert failure on heavily loaded store: Resolved
• [ABFS]: No GetPathStatus call for opening AbfsInputStream: Open
• [ABFS, S3A] Add IORateLimiter api to hadoop common: Open
• ABFS: Implement ThreadLocal for ObjectMapper in AzureHttpOperation via config option with static shared instance as an alternative.: Open
• [ABFS] Filesystem contract tests to use methodPath for robust parallel test runs: Open
• ABFS: Support FNS Accounts over BlobEndpoint: Open
• Improve ABFS metric integration with iOStatistics: Open
• ABFS: [FnsOverBlob]Making AbfsClient Abstract for supporting both DFS and Blob Endpoint: Resolved
• ABFS: [FnsOverBlob]Response Handling of Blob Endpoint APIs and Metadata APIs: Open
• ABFS: [FnsOverBlob]Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient: Open
• ABFS: [FnsOverBlob] Implementing Ingress Support with various Fallback Handling: Open
• ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint: Open
• ABFS: [FnsOverBlob] Adding Integration Tests for Special Scenarios in Blob Endpoint: Open
• ABFS: Allow "fs.azure.account.hns.enabled" to be set as Account Specific Config: Resolved
• ABFS: [FnsOverBlob] Group Enums in AbfsRestOperationType into categories based on Endpoint: Open