[HADOOP-15763] Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: fs/azure
• Assignee: Da Zhou
• Reporter: Steve Loughran
• Created: 17/Sep/18 22:19
• Updated: 02/Feb/21 16:17

Description

ABFS phase II: address issues which surface in the field; tune things which need tuning, add more tests where appropriate. Improve docs, especially troubleshooting. Classpaths. The usual.

Issue Links

• depends upon: HADOOP-17474 Optimise abfs incremental listings
• depends upon: HADOOP-15407 Support Windows Azure Storage - Blob file system in Hadoop
• relates to: HADOOP-16353 Backport ABFS changes from trunk to branch-3.2

Sub-Tasks

• Review/implement ABFS support for the extra fs ops which some apps (HBase) expects: Open
• ABFS: Increase timeout of ITestAbfsReadWriteAndSeek: Patch Available
• ABFS to support a Delegation Token provider which marshalls current Oauth secrets: Open
• Add ABFS support for Async Scatter/Gather IO: Open
• ABFS tests to include terasort: Open
• ABFS VersionedFileStatus to declare that it isEncrypted(): Patch Available
• ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side: Open
• ABFS: improve random read performance: Open
• abfs can't access storage account without password: Open
• ABFS: Send ReadAhead feature utilization metrics to server: Open
• ABFS Adding an option to Always use Read Ahead, even for non sequential reads: Patch Available
• Add support for Filesystem#getFileChecksum in ABFS driver: Open
• ABFS: Have all external dependent module execution tracked with DurationInfo: Open
• abfs can't access storage account if soft delete is enabled: Open
• ABFS: Fix for OutofMemoryException from AbfsOutputStream: Open
• ABFS: Support single writer dirs: Open
• Clear abfs readahead requests on stream close: Open
• ITestAbfsInputStreamStatistics#testReadAheadCounters timing out always: Reopened
• ABFS: Test failure ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics: Open
• ABFS: Random read perf improvement: Open
• ABFS: Test testAbfsStreamOps timing out: Open
• ABFS: Testcase failure ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics: Patch Available
• ABFS: Add Identifiers to Client Request Header: Open
• ABFS: Implementation for getContentSummary: Open
• Pass exception to the caller in AbfsClient#appendSASTokenToQuery: Open
• abfs ITestAzureBlobFileSystemCheckAccess test failure if test doesn't have oauth keys: Open
• ABFS: Suport for customer provided encrption key: Open
• ABFS test setup failing "The specified filesystem already exists": Open
• ABFS: Disable throttling update for auth failures: Open
• ABFS: Introduce Lease Operations with Append to provide single writer semantics: Open
• ABFS: Change default Readahead Queue Depth from num(processors) to const: Open
• ABFS: Partially obfuscate SAS object IDs in Logs: Open
• ABFS: Distcp contract test testDistCpWithIterator is timing out consistently: Open