Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15763

Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: fs/azure
    • Labels:
      None

      Description

      ABFS phase II: address issues which surface in the field; tune things which need tuning, add more tests where appropriate. Improve docs, especially troubleshooting. Classpaths. The usual.

        Attachments

        Issue Links

        1.
        ABFS checkException to map 403 to AccessDeniedException Sub-task Resolved Steve Loughran

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 3h 50m
        Actions
        2.
        intermittent failure of TestAbfsClient.validateUserAgent Sub-task Resolved Bilahari T H   Actions
        3.
        Review/implement ABFS support for the extra fs ops which some apps (HBase) expects Sub-task Open Unassigned   Actions
        4.
        ABFS: Increase timeout of ITestAbfsReadWriteAndSeek Sub-task Patch Available Bilahari T H

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        5.
        ABFS: Better exception handling of DNS errors Sub-task Resolved Da Zhou   Actions
        6.
        ABFS: validate packing, transient classpath, hadoop fs CLI Sub-task Resolved Steve Loughran   Actions
        7.
        Document ABFS Sub-task Resolved Thomas Marqardt   Actions
        8.
        typo in AzureBlobFileSystem.getIsNamespaceEnabeld Sub-task Resolved Abhishek Modi   Actions
        9.
        Make HTTPS the default protocol for ABFS Sub-task Resolved Da Zhou   Actions
        10.
        ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console Sub-task Resolved Da Zhou   Actions
        11.
        ABFS: Stop requiring client ID and tenant ID for MSI Sub-task Resolved Da Zhou   Actions
        12.
        ABFS: Enable some tests for namespace not enabled account using OAuth Sub-task Resolved Da Zhou   Actions
        13.
        ABFS: change createRemoteFileSystemDuringInitialization default to true Sub-task Resolved Sean Mackrory   Actions
        14.
        ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. Sub-task Resolved junhua gu   Actions
        15.
        ABFS: Throw IllegalArgumentException when Directory/File name ends with a period(.) Sub-task Resolved Shweta   Actions
        16.
        ABFS to support a Delegation Token provider which marshalls current Oauth secrets Sub-task Open Unassigned   Actions
        17.
        Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore Sub-task Resolved Da Zhou   Actions
        18.
        ABFS: make retry policy configurable Sub-task Resolved Da Zhou   Actions
        19.
        ABFS: For HNS account, avoid unnecessary get call when doing Rename Sub-task Resolved Da Zhou   Actions
        20.
        ABFS: replace $superuser with local user Sub-task Resolved Da Zhou   Actions
        21.
        ABFS: Enable owner and group conversion for MSI and login user using OAuth Sub-task Resolved Da Zhou   Actions
        22.
        ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 Sub-task Resolved junhua gu   Actions
        23.
        Add ABFS support for Async Scatter/Gather IO Sub-task Open Unassigned   Actions
        24.
        ABFS: add try catch for UGI failure when initializing ABFS Sub-task Resolved Da Zhou   Actions
        25.
        ABFS: getNamespaceEnabled can fail blocking user access thru ACLs Sub-task Resolved Da Zhou   Actions
        26.
        ABFS: reduce list page size to to 500 Sub-task Resolved Da Zhou   Actions
        27.
        ABFS: remove timeout check for DELETE and RENAME Sub-task Resolved Da Zhou   Actions
        28.
        ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException Sub-task Resolved Da Zhou   Actions
        29.
        ABFS: Bug fix for tolerateOobAppends configuration Sub-task Resolved Da Zhou   Actions
        30.
        UserAgent string for ABFS Sub-task Resolved Shweta   Actions
        31.
        ABFS: Better exception handling of DNS errors followup Sub-task Resolved Da Zhou   Actions
        32.
        ABFS: Fix Date format parser Sub-task Resolved Da Zhou   Actions
        33.
        ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store Sub-task Resolved Steve Loughran   Actions
        34.
        NativeAzureFileSystem does not support setXAttr Sub-task Resolved Clemens Wolff   Actions
        35.
        Failure of ABFS test ITestAbfsIdentityTransformer Sub-task Resolved Steve Loughran   Actions
        36.
        Wasb tests to downgrade to skip when test a/c is namespace enabled Sub-task Resolved Masatake Iwasaki   Actions
        37.
        ABFS: Should only transform username to short name Sub-task Resolved Da Zhou   Actions
        38.
        hadoop fs mkdir / of nonexistent abfs container raises NPE Sub-task Resolved Gabor Bota   Actions
        39.
        NPE in ABFS Client Credential Auth Sub-task Resolved Steve Loughran   Actions
        40.
        NPE in setup/teardown of ITestAbfsDelegationTokens Sub-task Resolved Steve Loughran   Actions
        41.
        ABFS: Bug fix for getPathProperties Sub-task Resolved Da Zhou   Actions
        42.
        Update abfs storage back-end with "close" flag when application is done writing to a file Sub-task Resolved Vishwajeet Dusane   Actions
        43.
        ABFS tests to include terasort Sub-task Open Unassigned   Actions
        44.
        ABFS VersionedFileStatus to declare that it isEncrypted() Sub-task Patch Available Masatake Iwasaki   Actions
        45.
        ABFS: add bufferpool to AbfsOutputStream Sub-task Resolved Da Zhou   Actions
        46.
        ABFS: add FSMainOperationsBaseTest Sub-task Resolved Da Zhou   Actions
        47.
        ABFS: add listFileStatus with StartFrom Sub-task Resolved Da Zhou   Actions
        48.
        ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side Sub-task Open Steve Loughran   Actions
        49.
        ABFS: transform full UPN for named user in AclStatus Sub-task Resolved Da Zhou   Actions
        50.
        ABFS: improve random read performance Sub-task Open Unassigned   Actions
        51.
        ABFS driver continues to retry on IOException responses from REST operations Sub-task Resolved Robert Levas   Actions
        52.
        ABFS: Override access() to no-op for now Sub-task Resolved Da Zhou   Actions
        53.
        ABFS: port Azure doc to 3.2 branch Sub-task Resolved Masatake Iwasaki   Actions
        54.
        ABFS default blocksize change(256MB from 512MB) Sub-task Resolved Arun Singh   Actions
        55.
        abfs can't access storage account without password Sub-task Open Masatake Iwasaki   Actions
        56.
        ABFS: Implement FileSystem.access() method Sub-task Resolved Bilahari T H   Actions
        57.
        ABFS FileStatus.getModificationTime returns localized time instead of UTC Sub-task Resolved Bilahari T H   Actions
        58.
        AzureADAuthenticator cannot authenticate in china Sub-task Resolved Unassigned   Actions
        59.
        Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property Sub-task Resolved Gabor Bota   Actions
        60.
        ABFS: Add missing query parameter for getPathStatus Sub-task Resolved Adam Antal   Actions
        61.
        ABFS: Config to enable/disable flush operation Sub-task Resolved Sneha Vijayarajan   Actions
        62.
        ABFS: Send ReadAhead feature utilization metrics to server Sub-task Open Sneha Vijayarajan   Actions
        63.
        ABFS: fileSystemExists() should not call container level apis Sub-task Resolved Sneha Vijayarajan   Actions
        64.
        Make AAD endpoint configurable on all Auth flows Sub-task Resolved Bilahari T H   Actions
        65.
        Track Azure Blob File System client-perceived latency Sub-task Resolved Jeetesh Mangwani   Actions
        66.
        Backport HADOOP-16548 - "ABFS: Config to enable/disable flush operation" to branch-2 Sub-task Resolved Sneha Vijayarajan   Actions
        67.
        Backport HADOOP-16578 - "ABFS: fileSystemExists() should not call container level apis" to Branch-2 Sub-task Resolved Sneha Vijayarajan   Actions
        68.
        Backport HADOOP-16587 - "Make AAD endpoint configurable on all Auth flows" to branch-2 Sub-task Resolved Bilahari T H   Actions
        69.
        ABFS: add missing docs for configuration Sub-task Resolved Bilahari T H   Actions
        70.
        ABFS: Make RetryCount in ExponentialRetryPolicy Configurable Sub-task Resolved Sneha Vijayarajan   Actions
        71.
        Remove unnecessary ABFS toString() invocations Sub-task Resolved Bilahari T H   Actions
        72.
        ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account Sub-task Resolved Sneha Vijayarajan   Actions
        73.
        ABFS Adding an option to Always use Read Ahead, even for non sequential reads Sub-task Patch Available Saurabh   Actions
        74.
        ABFS: Enhance driver debug logs Sub-task Resolved Sneha Vijayarajan   Actions
        75.
        ABFS: Add additional tests for CustomTokenProvider Sub-task Resolved Sneha Vijayarajan   Actions
        76.
        ITestClientUrlScheme fails for accounts which don't support HTTP Sub-task Resolved Steve Loughran   Actions
        77.
        NPE in UGI.getCurrentUser in ITestAbfsIdentityTransformer setup Sub-task Resolved Steve Loughran   Actions
        78.
        Backport HADOOP-16612 - "Track Azure Blob File System client-perceived latency" to branch-2 Sub-task Resolved Bilahari T H   Actions
        79.
        Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to branch-2 Sub-task Resolved Bilahari T H   Actions
        80.
        Backport HADOOP-16548 - ABFS: Config to enable/disable flush operation issue to branch-3.2 Sub-task Resolved Sneha Vijayarajan   Actions
        81.
        ABFS: test failure testLastModifiedTime() Sub-task Resolved Sneha Vijayarajan   Actions
        82.
        Add support for Filesystem#getFileChecksum in ABFS driver Sub-task Open Unassigned   Actions
        83.
        ABFS: Backport HADOOP-16660 ABFS: Make RetryCount in ExponentialRetryPolicy Configurable to Branch-2 Sub-task Resolved Sneha Vijayarajan   Actions
        84.
        Backport HADOOP-16612 "Track Azure Blob File System client-perceived latency" to branch-2 Sub-task Resolved Jeetesh Mangwani   Actions
        85.
        Improve wasb and abfs resilience on double close() calls Sub-task Resolved Steve Loughran   Actions
        86.
        ABFS: Have all external dependent module execution tracked with DurationInfo Sub-task Open Sneha Vijayarajan   Actions
        87.
        ABFS: Combine append+flush calls for blockblob & appendblob Sub-task Resolved Ishani   Actions
        88.
        ITestAzureBlobFileSystemCheckAccess failing Sub-task Resolved Bilahari T H   Actions
        89.
        ABFS: update abfs.md to include config keys for identity transformation Sub-task Resolved Karthick Narendran   Actions
        90.
        abfs can't access storage account if soft delete is enabled Sub-task Open Unassigned   Actions
        91.
        ITestAbfsClient.testContinuationTokenHavingEqualSign failing Sub-task Resolved Sneha Vijayarajan   Actions
        92.
        ABFS: Send error back to client for Read Ahead request failure Sub-task Resolved Sneha Vijayarajan   Actions
        93.
        ABFS: Fix for OutofMemoryException from AbfsOutputStream Sub-task Open Bilahari T H   Actions
        94.
        ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure Sub-task Resolved Bilahari T H   Actions
        95.
        ABFS: Optimize HttpRequest retry triggers Sub-task Resolved Sneha Vijayarajan   Actions
        96.
        ABFS: Add unbuffer support to AbfsInputStream Sub-task Resolved Sahil Takiar   Actions
        97.
        ABFS: Change in expiry calculation for MSI token provider Sub-task Resolved Bilahari T H   Actions
        98.
        ABFS: Support for OAuth v2.0 endpoints Sub-task Resolved Bilahari T H   Actions
        99.
        ABFS: Test failure ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance Sub-task Resolved Bilahari T H   Actions
        100.
        ABFS: Make list page size configurable Sub-task Resolved Bilahari T H   Actions
        101.
        ABFS: Change in User-Agent header Sub-task Resolved Bilahari T H   Actions
        102.
        Backport HADOOP-16890- "ABFS: Change in expiry calculation for MSI token provider" & HADOOP-16825 "ITestAzureBlobFileSystemCheckAccess failing" to branch-2 Sub-task Resolved Bilahari T H   Actions
        103.
        ABFS: Support single writer dirs Sub-task Open Billie Rinaldi

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 10h 50m
        Actions
        104.
        ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) Sub-task Resolved Mehakmeet Singh   Actions
        105.
        ABFS: Upgrade Store REST API Version to 2019-12-12 Sub-task Resolved Sneha Vijayarajan   Actions
        106.
        ABFS: Avoid storage calls to check if the account is HNS enabled or not Sub-task Resolved Bilahari T H   Actions
        107.
        ABFS: Improve the ABFS driver documentation Sub-task Resolved Bilahari T H   Actions
        108.
        ABFS: Make PUT and POST operations idempotent Sub-task Resolved Sneha Vijayarajan   Actions
        109.
        Support disabling buffered reads in ABFS positional reads Sub-task Resolved Anoop Sam John

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 9h 10m
        Actions
        110.
        ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting Sub-task Resolved Sneha Vijayarajan   Actions
        111.
        ABFS: Fix idempotency test failures when SharedKey is set as AuthType Sub-task Resolved Sneha Vijayarajan   Actions
        112.
        Support for Appendblob in abfs driver Sub-task Resolved Ishani   Actions
        113.
        javadoc failing in the yetus report with the latest trunk Sub-task Resolved Unassigned   Actions
        114.
        ABFS: Fix the parsing errors in ABFS Driver with creation Time (being returned in ListPath) Sub-task Resolved Bilahari T H   Actions
        115.
        ABFS: Long waits and unintended retries when multiple threads try to fetch token using ClientCreds Sub-task Resolved Bilahari T H   Actions
        116.
        ABFS: GetAccessToken unrecoverable failures are being retried Sub-task Resolved Bilahari T H   Actions
        117.
        ABFS: Fix For Idempotency code Sub-task Resolved Sneha Vijayarajan   Actions
        118.
        ABFS: Tests ITestAbfsNetworkStatistics need to be config setting agnostic Sub-task Resolved Bilahari T H   Actions
        119.
        ABFS: Test failure: testFailedRequestWhenCredentialsNotCorrect fails when run with SharedKey Sub-task Resolved Bilahari T H   Actions
        120.
        ABFS: Test failure: Disable ITestAzureBlobFileSystemDelegationSAS tests Sub-task Resolved Bilahari T H   Actions
        121.
        Clear abfs readahead requests on stream close Sub-task Open Unassigned   Actions
        122.
        ITestAbfsInputStreamStatistics#testReadAheadCounters timing out always Sub-task Reopened Steve Loughran   Actions
        123.
        ABFS: Add debug log for rename failures Sub-task Resolved Bilahari T H   Actions
        124.
        ABFS: configure output stream thread pool Sub-task Resolved Bilahari T H

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h
        Actions
        125.
        ABFS: Enable checkaccess API Sub-task Resolved Bilahari T H

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        Actions
        126.
        ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results Sub-task Resolved Bilahari T H

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 14h
        Actions
        127.
        ABFS: Test failure ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics Sub-task Open Unassigned   Actions
        128.
        ABFS: Support for conditional overwrite Sub-task Resolved Sneha Vijayarajan   Actions
        129.
        ABFS: Random read perf improvement Sub-task Open Mukund Thakur

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h
        Actions
        130.
        ABFS: Test testAbfsStreamOps timing out Sub-task Open Sneha Vijayarajan   Actions
        131.
        ABFS: Support for Client Correlation ID Sub-task Resolved Unassigned

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 3h 40m
        Actions
        132.
        ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account Sub-task Resolved Sneha Vijayarajan

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        Actions
        133.
        ABFS: Testcase failure ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics Sub-task Patch Available Unassigned

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        134.
        ABFS: Add Identifiers to Client Request Header Sub-task Open Sumangala Patki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h 20m
        Actions
        135.
        ABFS: Allow Random Reads to be of Buffer Size Sub-task Resolved Sneha Vijayarajan

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        Actions
        136.
        ABFS: read-ahead error reporting breaks buffer management Sub-task Resolved Sneha Vijayarajan

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 3h 50m
        Actions
        137.
        ABFS: Logs should redact SAS signature Sub-task Resolved Bilahari T H

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 3h 10m
        Actions
        138.
        Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-3.2 Sub-task Resolved Sally Zuo   Actions
        139.
        Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" and HADOOP-16785. "Improve wasb and abfs resilience on double close() calls. followup to abfs close() fix." to branch-2.10 Sub-task Resolved Sally Zuo   Actions
        140.
        ABFS: Optimise read for small files/tails of files Sub-task Resolved Bilahari T H

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 12.5h
        Actions
        141.
        ABFS: testRenameFileOverExistingFile Fails after Contract test update Sub-task Resolved Sneha Vijayarajan

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        Actions
        142.
        ABFS: SAS Test updates for version and permission update Sub-task Resolved Sneha Vijayarajan

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 50m
        Actions
        143.
        ABFS: Piggyback flush on Append calls for short writes Sub-task Resolved Sneha Vijayarajan

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 4h
        Actions
        144.
        ABFS: Delete Idempotency handling can lead to NPE Sub-task Resolved Sneha Vijayarajan

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 40m
        Actions
        145.
        ABFS: Release Elastic ByteBuffer pool memory at outputStream close Sub-task Resolved Sneha Vijayarajan

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        146.
        ABFS: Set default ListMaxResults to max server limit Sub-task Resolved Thomas Marqardt

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h 20m
        Actions
        147.
        ABFS: Implementation for getContentSummary Sub-task Open Sumangala Patki   Actions
        148.
        Optimise abfs incremental listings Sub-task Resolved Unassigned   Actions
        149.
        ABFS: Fix auth failures getting counted as throttling errors in ABFS throttling interceptor Sub-task Resolved Unassigned   Actions
        150.
        Pass exception to the caller in AbfsClient#appendSASTokenToQuery Sub-task Open Mukund Thakur   Actions
        151.
        ABFS: Fix boundary conditions in InputStream seek and skip Sub-task Resolved Unassigned

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 4h 50m
        Actions
        152.
        abfs ITestAzureBlobFileSystemCheckAccess test failure if test doesn't have oauth keys Sub-task Open Unassigned   Actions
        153.
        ABFS: Suport for customer provided encrption key Sub-task Open Bilahari T H

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 10h 10m
        Actions
        154.
        Correct abfs test assertion reversed in HADOOP-13327 Sub-task Resolved Sumangala Patki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        Actions
        155.
        ABFS test setup failing "The specified filesystem already exists" Sub-task Open Unassigned   Actions
        156.
        ABFS: Toggle Store Mkdirs request overwrite parameter Sub-task Resolved Sumangala Patki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 50m
        Actions
        157.
        ABFS: Disable throttling update for auth failures Sub-task Open Sumangala Patki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 3h
        Actions
        158.
        ABFS: Introduce Lease Operations with Append to provide single writer semantics Sub-task Open Sneha Varma   Actions
        159.
        ABFS: Change default Readahead Queue Depth from num(processors) to const Sub-task Open Sumangala Patki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        160.
        ABFS: Partially obfuscate SAS object IDs in Logs Sub-task Open Sumangala Patki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2.5h
        Actions
        161.
        ABFS: Distcp contract test testDistCpWithIterator is timing out consistently Sub-task Open Unassigned   Actions

          Activity

            People

            • Assignee:
              DanielZhou Da Zhou
              Reporter:
              stevel@apache.org Steve Loughran

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 106h 40m
                106h 40m

                  Issue deployment