Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15763

Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: fs/azure
    • Labels:
      None

      Description

      ABFS phase II: address issues which surface in the field; tune things which need tuning, add more tests where appropriate. Improve docs, especially troubleshooting. Classpaths. The usual.

        Attachments

          Issue Links

          1.
          ABFS checkException to map 403 to AccessDeniedException Sub-task Open Bilahari T H
          2.
          intermittent failure of TestAbfsClient.validateUserAgent Sub-task Resolved Bilahari T H
          3.
          Review/implement ABFS support for the extra fs ops which some apps (HBase) expects Sub-task Open Unassigned
          4.
          ABFS: Increase timeout of ITestAbfsReadWriteAndSeek Sub-task Patch Available Bilahari T H
          5.
          ABFS: Better exception handling of DNS errors Sub-task Resolved Da Zhou
          6.
          ABFS: validate packing, transient classpath, hadoop fs CLI Sub-task Resolved Steve Loughran
          7.
          Document ABFS Sub-task Resolved Thomas Marqardt
          8.
          typo in AzureBlobFileSystem.getIsNamespaceEnabeld Sub-task Resolved Abhishek Modi
          9.
          Make HTTPS the default protocol for ABFS Sub-task Resolved Da Zhou
          10.
          ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console Sub-task Resolved Da Zhou
          11.
          ABFS: Stop requiring client ID and tenant ID for MSI Sub-task Resolved Da Zhou
          12.
          ABFS: Enable some tests for namespace not enabled account using OAuth Sub-task Resolved Da Zhou
          13.
          ABFS: change createRemoteFileSystemDuringInitialization default to true Sub-task Resolved Sean Mackrory
          14.
          ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. Sub-task Resolved junhua gu
          15.
          ABFS: Throw IllegalArgumentException when Directory/File name ends with a period(.) Sub-task Resolved Shweta
          16.
          ABFS to support a Delegation Token provider which marshalls current Oauth secrets Sub-task Open Unassigned
          17.
          Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore Sub-task Resolved Da Zhou
          18.
          ABFS: make retry policy configurable Sub-task Resolved Da Zhou
          19.
          ABFS: For HNS account, avoid unnecessary get call when doing Rename Sub-task Resolved Da Zhou
          20.
          ABFS: replace $superuser with local user Sub-task Resolved Da Zhou
          21.
          ABFS: Enable owner and group conversion for MSI and login user using OAuth Sub-task Resolved Da Zhou
          22.
          ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 Sub-task Resolved junhua gu
          23.
          Add ABFS support for Async Scatter/Gather IO Sub-task Open Unassigned
          24.
          ABFS: add try catch for UGI failure when initializing ABFS Sub-task Resolved Da Zhou
          25.
          ABFS: getNamespaceEnabled can fail blocking user access thru ACLs Sub-task Resolved Da Zhou
          26.
          ABFS: reduce list page size to to 500 Sub-task Resolved Da Zhou
          27.
          ABFS: remove timeout check for DELETE and RENAME Sub-task Resolved Da Zhou
          28.
          ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException Sub-task Resolved Da Zhou
          29.
          ABFS: Bug fix for tolerateOobAppends configuration Sub-task Resolved Da Zhou
          30.
          UserAgent string for ABFS Sub-task Resolved Shweta
          31.
          ABFS: Better exception handling of DNS errors followup Sub-task Resolved Da Zhou
          32.
          ABFS: Fix Date format parser Sub-task Resolved Da Zhou
          33.
          ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store Sub-task Resolved Steve Loughran
          34.
          NativeAzureFileSystem does not support setXAttr Sub-task Resolved Clemens Wolff
          35.
          Failure of ABFS test ITestAbfsIdentityTransformer Sub-task Resolved Steve Loughran
          36.
          Wasb tests to downgrade to skip when test a/c is namespace enabled Sub-task Resolved Masatake Iwasaki
          37.
          ABFS: Should only transform username to short name Sub-task Resolved Da Zhou
          38.
          hadoop fs mkdir / of nonexistent abfs container raises NPE Sub-task Resolved Gabor Bota
          39.
          NPE in ABFS Client Credential Auth Sub-task Resolved Steve Loughran
          40.
          NPE in setup/teardown of ITestAbfsDelegationTokens Sub-task Resolved Steve Loughran
          41.
          ABFS: Bug fix for getPathProperties Sub-task Resolved Da Zhou
          42.
          Update abfs storage back-end with "close" flag when application is done writing to a file Sub-task Resolved Vishwajeet Dusane
          43.
          ABFS tests to include terasort Sub-task Open Unassigned
          44.
          ABFS VersionedFileStatus to declare that it isEncrypted() Sub-task Patch Available Masatake Iwasaki
          45.
          ABFS: add bufferpool to AbfsOutputStream Sub-task Resolved Da Zhou
          46.
          ABFS: add FSMainOperationsBaseTest Sub-task Resolved Da Zhou
          47.
          ABFS: add listFileStatus with StartFrom Sub-task Resolved Da Zhou
          48.
          ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side Sub-task Open Steve Loughran
          49.
          ABFS: transform full UPN for named user in AclStatus Sub-task Resolved Da Zhou
          50.
          ABFS: improve random read performance Sub-task Open Unassigned
          51.
          ABFS driver continues to retry on IOException responses from REST operations Sub-task Resolved Robert Levas
          52.
          ABFS: Override access() to no-op for now Sub-task Resolved Da Zhou
          53.
          ABFS: port Azure doc to 3.2 branch Sub-task Resolved Masatake Iwasaki
          54.
          ABFS default blocksize change(256MB from 512MB) Sub-task Resolved Arun Singh
          55.
          abfs can't access storage account without password Sub-task Open Masatake Iwasaki
          56.
          ABFS: Implement FileSystem.access() method Sub-task Resolved Bilahari T H
          57.
          ABFS FileStatus.getModificationTime returns localized time instead of UTC Sub-task Resolved Bilahari T H
          58.
          AzureADAuthenticator cannot authenticate in china Sub-task Resolved Unassigned
          59.
          Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property Sub-task Resolved Gabor Bota
          60.
          ABFS: Add missing query parameter for getPathStatus Sub-task Resolved Adam Antal
          61.
          ABFS: Config to enable/disable flush operation Sub-task Resolved Sneha Vijayarajan
          62.
          ABFS: Send ReadAhead feature utilization metrics to server Sub-task Open Sneha Vijayarajan
          63.
          ABFS: fileSystemExists() should not call container level apis Sub-task Resolved Sneha Vijayarajan
          64.
          Make AAD endpoint configurable on all Auth flows Sub-task Resolved Bilahari T H
          65.
          Track Azure Blob File System client-perceived latency Sub-task Resolved Jeetesh Mangwani
          66.
          Backport HADOOP-16548 - "ABFS: Config to enable/disable flush operation" to branch-2 Sub-task Resolved Sneha Vijayarajan
          67.
          Backport HADOOP-16578 - "ABFS: fileSystemExists() should not call container level apis" to Branch-2 Sub-task Resolved Sneha Vijayarajan
          68.
          Backport HADOOP-16587 - "Make AAD endpoint configurable on all Auth flows" to branch-2 Sub-task Resolved Bilahari T H
          69.
          ABFS: add missing docs for configuration Sub-task Resolved Bilahari T H
          70.
          ABFS: Make RetryCount in ExponentialRetryPolicy Configurable Sub-task Resolved Sneha Vijayarajan
          71.
          Remove unnecessary ABFS toString() invocations Sub-task Resolved Bilahari T H
          72.
          ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account Sub-task Resolved Sneha Vijayarajan
          73.
          ABFS Adding an option to Always use Read Ahead, even for non sequential reads Sub-task Patch Available Saurabh
          74.
          ABFS: Enhance driver debug logs Sub-task Resolved Sneha Vijayarajan
          75.
          ABFS: Add additional tests for CustomTokenProvider Sub-task Resolved Sneha Vijayarajan
          76.
          ITestClientUrlScheme fails for accounts which don't support HTTP Sub-task Resolved Steve Loughran
          77.
          NPE in UGI.getCurrentUser in ITestAbfsIdentityTransformer setup Sub-task Resolved Steve Loughran
          78.
          Backport HADOOP-16612 - "Track Azure Blob File System client-perceived latency" to branch-2 Sub-task Resolved Bilahari T H
          79.
          Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to branch-2 Sub-task Resolved Bilahari T H
          80.
          Backport HADOOP-16548 - ABFS: Config to enable/disable flush operation issue to branch-3.2 Sub-task Resolved Sneha Vijayarajan
          81.
          ABFS: test failure testLastModifiedTime() Sub-task Resolved Sneha Vijayarajan
          82.
          Add support for Filesystem#getFileChecksum in ABFS driver Sub-task Open Unassigned
          83.
          ABFS: Backport HADOOP-16660 ABFS: Make RetryCount in ExponentialRetryPolicy Configurable to Branch-2 Sub-task Resolved Sneha Vijayarajan
          84.
          Backport HADOOP-16612 "Track Azure Blob File System client-perceived latency" to branch-2 Sub-task Resolved Jeetesh Mangwani
          85.
          Improve wasb and abfs resilience on double close() calls Sub-task Resolved Steve Loughran
          86.
          ABFS: Have all external dependent module execution tracked with DurationInfo Sub-task Open Sneha Vijayarajan
          87.
          ABFS: Combine append+flush calls for blockblob & appendblob Sub-task Resolved Ishani
          88.
          ITestAzureBlobFileSystemCheckAccess failing Sub-task Resolved Bilahari T H
          89.
          ABFS: update abfs.md to include config keys for identity transformation Sub-task Resolved Karthick Narendran
          90.
          abfs can't access storage account if soft delete is enabled Sub-task Open Unassigned
          91.
          ITestAbfsClient.testContinuationTokenHavingEqualSign failing Sub-task Resolved Sneha Vijayarajan
          92.
          ABFS: Send error back to client for Read Ahead request failure Sub-task Resolved Sneha Vijayarajan
          93.
          ABFS: Fix for OutofMemoryException from AbfsOutputStream Sub-task Open Bilahari T H
          94.
          ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure Sub-task Resolved Bilahari T H
          95.
          ABFS: Optimize HttpRequest retry triggers Sub-task Resolved Sneha Vijayarajan
          96.
          ABFS: Add unbuffer support to AbfsInputStream Sub-task Resolved Sahil Takiar
          97.
          ABFS: Change in expiry calculation for MSI token provider Sub-task Resolved Bilahari T H
          98.
          ABFS: Support for OAuth v2.0 endpoints Sub-task Resolved Bilahari T H
          99.
          ABFS: Test failure ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance Sub-task Open Bilahari T H
          100.
          ABFS: Make list page size configurable Sub-task Resolved Bilahari T H
          101.
          ABFS: Change in User-Agent header Sub-task Resolved Bilahari T H
          102.
          Backport HADOOP-16890- "ABFS: Change in expiry calculation for MSI token provider" & HADOOP-16825 "ITestAzureBlobFileSystemCheckAccess failing" to branch-2 Sub-task Resolved Bilahari T H
          103.
          ABFS: Support single writer dirs Sub-task Open Billie Rinaldi
          104.
          ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) Sub-task Resolved Mehakmeet Singh
          105.
          ABFS: Upgrade Store REST API Version to 2019-12-12 Sub-task Patch Available Sneha Vijayarajan
          106.
          ABFS: Avoid storage calls to check if the account is HNS enabled or not Sub-task Resolved Bilahari T H
          107.
          ABFS: Improve the ABFS driver documentation Sub-task Resolved Bilahari T H
          108.
          ABFS: Make PUT and POST operations idempotent Sub-task Resolved Sneha Vijayarajan
          109.
          Support positional read in AbfsInputStream Sub-task Open Anoop Sam John
          110.
          ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting Sub-task Resolved Sneha Vijayarajan
          111.
          ABFS: Fix idempotency test failures when SharedKey is set as AuthType Sub-task Resolved Sneha Vijayarajan
          112.
          Support for Appendblob in abfs driver Sub-task Resolved Ishani
          113.
          javadoc failing in the yetus report with the latest trunk Sub-task Resolved Unassigned
          114.
          ABFS: Fix the parsing errors in ABFS Driver with creation Time (being returned in ListPath) Sub-task Resolved Bilahari T H
          115.
          ABFS: Long waits and unintended retries when multiple threads try to fetch token using ClientCreds Sub-task Resolved Bilahari T H
          116.
          ABFS: GetAccessToken unrecoverable failures are being retried Sub-task Resolved Bilahari T H
          117.
          ABFS: Fix For Idempotency code Sub-task Resolved Sneha Vijayarajan
          118.
          ABFS: Tests ITestAbfsNetworkStatistics need to be config setting agnostic Sub-task Resolved Bilahari T H
          119.
          ABFS: Test failure: testFailedRequestWhenCredentialsNotCorrect fails when run with SharedKey Sub-task Resolved Bilahari T H
          120.
          ABFS: Test failure: Disable ITestAzureBlobFileSystemDelegationSAS tests Sub-task Resolved Bilahari T H
          121.
          Clear abfs readahead requests on stream close Sub-task Open Unassigned
          122.
          ITestAbfsInputStreamStatistics#testReadAheadCounters timing out always Sub-task Resolved Steve Loughran
          123.
          ABFS: Add debug log for rename failures Sub-task Resolved Bilahari T H
          124.
          ABFS: Add configs for maxConcurrentRequestCount and threadpool queue size Sub-task Patch Available Bilahari T H
          125.
          ABFS: Enable checkaccess API Sub-task Resolved Bilahari T H
          126.
          ABFS: Run tests with all the auth types Sub-task Open Bilahari T H

            Activity

              People

              • Assignee:
                DanielZhou Da Zhou
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated: