Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-19092

ABFS phase 4: post Hadoop 3.4.0 features

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • fs/azure
    • None

    Description

      Uber-JIRA for ABFS work so we can close HADOOP-18072 as done for 3.4.0

      Assuming 3.4.1 is a rapid roll of packing, dependencies and critical fixes, this should target 3.4.2 and beyond

      Attachments

        Issue Links

          1.
          Fix for Javac Error of Unchecked Cast Sub-task Open Anuj Modi  
          2.
          ABFS: Fix FileSystemAlreadyExists error for AzureAD errors Sub-task Open Anmol Asrani  
          3.
          ABFS: Toggle Store Mkdirs request overwrite parameter with default value Sub-task Open Anmol Asrani  
          4.
          ABFS: Modifications to TracingContext Sub-task Open Unassigned  
          5.
          ABFS: Add sendMs and recvMs for all AbfsHttpOperation calls Sub-task Open Anmol Asrani  
          6.
          ABFS:Support for secondary accounts on ABFS Driver for SharedKey Auth Sub-task Open Anmol Asrani  
          7.
          ITestAbfsInputStreamStatistics#testReadAheadCounters timing out always Sub-task Reopened Steve Loughran  
          8.
          ABFS: ITestAzureBlobFileSystemLease test failure for Appendblob HNS OAuth Sub-task Open Unassigned  
          9.
          FileNotFoundException in abfs mkdirs() call Sub-task Open Sneha Vijayarajan  
          10.
          Pass exception to the caller in AbfsClient#appendSASTokenToQuery Sub-task Open Mukund Thakur  
          11.
          abfs testReadAndWriteWithDifferentBufferSizesAndSeek failure Sub-task Open Unassigned  
          12.
          [ABFS][Retry Policy] Using hadoop-common code to refractor Abfs Retry Policy Implementation Sub-task Open Anuj Modi  
          13.
          ABFS: Test testAbfsStreamOps timing out Sub-task Open Sneha Vijayarajan  
          14.
          ABFS: Add support for cache handling when filesystem instance has clientCorrelationID enabled Sub-task Open Anmol Asrani

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          15.
          ABFS: delete call sets Socket timeout lesser than query timeout leading to failures Sub-task Open Unassigned  
          16.
          ABFS test setup failing "The specified filesystem already exists" Sub-task Open Unassigned  
          17.
          ABFS: Fix for OutofMemoryException from AbfsOutputStream Sub-task Open Bilahari T H  
          18.
          ABFS: Improve test scripts Sub-task Patch Available Sneha Vijayarajan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          19.
          abfs can't access storage account if soft delete is enabled Sub-task Open Unassigned  
          20.
          ABFS Adding an option to Always use Read Ahead, even for non sequential reads Sub-task Patch Available Saurabh  
          21.
          ABFS to support a Delegation Token provider which marshalls current Oauth secrets Sub-task Open Unassigned  
          22.
          [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout failures Sub-task Resolved Anuj Modi  
          23.
          transient failure of ITestAbfsListStatusRemoteIterator.testWithAbfsIteratorDisabledWithoutHasNext Sub-task Open Unassigned  
          24.
          Abfs directory delete times out on large directory tree w/ Oauth: OperationTimedOut Sub-task Open Unassigned  
          25.
          ABFS: Testcase failure ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics Sub-task Patch Available Unassigned

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          26.
          ABFS: Support for Pagination in Recursive Directory Delete Sub-task In Progress Anuj Modi  
          27.
          Add support for Filesystem#getFileChecksum in ABFS driver Sub-task Open Unassigned  
          28.
          ABFS: improve random read performance Sub-task Open Unassigned  
          29.
          ABFS VersionedFileStatus to declare that it isEncrypted() Sub-task Patch Available Masatake Iwasaki  
          30.
          ABFS: Implementation for getContentSummary Sub-task Open Sumangala Patki

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 40m
          31.
          ABFS: AbfsOutputStream doesnt close DataBlocks object. Sub-task Open Pranav Saxena  
          32.
          AbstractAbfsIntegrationTest.createFilesystemForSASTests() leaks FS instances Sub-task Open Steve Loughran  
          33.
          WASB: Fix Compiler Warnings Sub-task Open Unassigned  
          34.
          ABFS: Introduce Lease Operations with Append to provide single writer semantics Sub-task Open Sneha Varma

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m
          35.
          ABFS: ITestAbfsStreamStatistics TestAbfsStreamOps fail with append blob on HNS account Sub-task Open Unassigned  
          36.
          ABFS: Add correlated metric support for ABFS operations Sub-task Resolved Anmol Asrani  
          37.
          ABFS: rename ListResultSchemaTest to TestListResultSchema so maven tests run it Sub-task Open Anoop Sam John  
          38.
          ABFS: Refactor HTTP request handling code Sub-task Open Sneha Vijayarajan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h
          39.
          Add ABFS support for Async Scatter/Gather IO Sub-task Open Anmol Asrani  
          40.
          ABFS IsNamespaceEnabled shouldn't use/log exceptions as default execution path Sub-task Open Unassigned  
          41.
          ABFS: Lease operations Sub-task Open Anmol Asrani  
          42.
          ABFS: Fork AbfsHttpOperation to add alternate connection Sub-task Open Sneha Vijayarajan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 50m
          43.
          ABFS: Enable optional store connectivity over azure specific protocol for data egress Sub-task Open Sneha Vijayarajan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          44.
          ABFS: Refactor read flow to include ReadRequestParameter Sub-task Open Sneha Vijayarajan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          45.
          ABFS: test failure testLastModifiedTime() Sub-task Reopened Sneha Vijayarajan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          46.
          ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side Sub-task Open Steve Loughran  
          47.
          ABFS: Enable config control for default connection timeout Sub-task Resolved Sneha Vijayarajan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          48.
          ABFS to log @ debug each config option *And where it comes from* Sub-task Open Unassigned  
          49.
          ABFS AbfsDelegationTokenManager to generate canonicalServiceName if DT plugin doesn't Sub-task Open Steve Loughran

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          50.
          ABFS: transient failure of TestAzureBlobFileSystemFileStatus.testLastModifiedTime Sub-task Open Anoop Sam John  
          51.
          ABFS: Send ReadAhead feature utilization metrics to server Sub-task Open Sneha Vijayarajan  
          52.
          [ABFS] Reverting Back Support of setXAttr() and getXAttr() on root path Sub-task Resolved Anuj Modi  
          53.
          ABFS: Have all external dependent module execution tracked with DurationInfo Sub-task Open Sneha Vijayarajan  
          54.
          ABFS: Customize and optimize timeouts made based on each separate request Sub-task Open Sree Bhattacharyya  
          55.
          ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException Sub-task Reopened Sumangala Patki

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 7h 20m
          56.
          Review/implement ABFS support for the extra fs ops which some apps (HBase) expects Sub-task Open Unassigned  
          57.
          ABFS: Remove the workaround used for Http PATCH Sub-task Open Unassigned  
          58.
          ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting Sub-task Open Unassigned  
          59.
          ABFS: Enabling Client-side Backoff only for new requests Sub-task Open Sree Bhattacharyya  
          60.
          ABFS contract-tests with Hadoop-Commons intermittently failing Sub-task Open Unassigned  
          61.
          ABFS: Remove commons IOUtils.close() from AbfsOutputStream Sub-task Open Mehakmeet Singh  
          62.
          ABFS: Skip testEtagConsistencyAcrossRename for Non-HNS accounts Sub-task Open Unassigned

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          63.
          ABFS: Increase timeout of ITestAbfsReadWriteAndSeek Sub-task Patch Available Bilahari T H

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          64.
          ABFS: Test with 100MB buffer size in ITestAbfsReadWriteAndSeek times out Sub-task Open Sneha Vijayarajan  
          65.
          abfs can't access storage account without password Sub-task Open Masatake Iwasaki  
          66.
          [ABFS] Support VectorIO in ABFS Input Stream Sub-task Open Anmol Asrani  
          67.
          [ABFS]: Support fixed SAS token config in addition to Custom SASTokenProvider Implementation Sub-task Resolved Anuj Modi  
          68.
          Improve rate limiting through ABFS in Manifest Committer Sub-task Open Steve Loughran  
          69.
          [ABFS] Enhancing Client-Side Throttling Metrics Updation Logic Sub-task Resolved Anuj Modi  
          70.
          [ABFS]: FooterReadBufferSize should not be greater than readBufferSize Sub-task Resolved Pranav Saxena  
          71.
          [ABFS]: ApacheHttpClient adaptation as network library Sub-task Open Pranav Saxena  
          72.
          [ABFS] testListPathWithValueGreaterThanServerMaximum assert failure on heavily loaded store Sub-task Open Anuj Modi  
          73.
          ABFS: Fixing Test Script Bug and Some Known test Failures in ABFS Test Suite Sub-task Resolved Anuj Modi  
          74.
          [ABFS]Prevent ABFS initialization for non-hierarchical-namespace account if Customer-provided-key configs given. Sub-task Resolved Pranav Saxena  
          75.
          [ABFS]: No GetPathStatus call for opening AbfsInputStream Sub-task Open Pranav Saxena  
          76.
          [ABFS, S3A] Add IORateLimiter api to hadoop common Sub-task In Progress Steve Loughran  
          77.
          ABFS: Implement ThreadLocal for ObjectMapper in AzureHttpOperation via config option with static shared instance as an alternative. Sub-task Open Mukund Thakur  
          78.
          Test ITestAbfsRestOperationException#testAuthFailException is broken. Sub-task Resolved Anuj Modi  
          79.
          [ABFS] Filesystem contract tests to use methodPath for robust parallel test runs Sub-task Open Steve Loughran  
          80.
          WASB Driver Deprecation and eventual removal Sub-task Resolved Anuj Modi  
          81.
          ABFS: Support FNS Accounts over BlobEndpoint Sub-task Open Sneha Vijayarajan  
          82.
          Improve ABFS metric integration with iOStatistics Sub-task Open Unassigned  
          83.
          ABFS: Making AbfsClient Abstract for supporting both DFS and Blob Endpoint Sub-task Open Anuj Modi  

          Activity

            People

              snvijaya Sneha Vijayarajan
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 30h
                  30h