Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14552

Über-jira: WASB client phase II: performance and testing

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.1
    • 2.9.0, 3.0.0
    • fs/azure
    • None

    Description

      Uber-JIRA for wasb:// phase II: the things we want for Hadoop 2.9.

      I think the focus is on performance, but looking at the tests there are some things we need there first: parallel test execution, move over to IT tests, other tuning.

      All patches must come with declarations of which azure endpoint they were tested against.

      Attachments

        Issue Links

        1.
        Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Sub-task Resolved Georgi Chalakov Actions
        2.
        Upgrade azure-storage sdk version >5.4.0 Sub-task Resolved Rajesh Balamohan Actions
        3.
        Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients Sub-task Closed Unassigned Actions
        4.
        Remove service loader config file for wasb fs Sub-task Resolved Esfandiar Manii Actions
        5.
        Add (parallelized) integration tests to hadoop-azure Sub-task Resolved Steve Loughran Actions
        6.
        TestFileSystemOperationExceptionMessage to not rerun all of NativeAzureFileSystemBaseTest Sub-task Resolved Unassigned Actions
        7.
        document how to run wasb tests in azure docs site/testing.md Sub-task Resolved Unassigned Actions
        8.
        [WASB] the configured retry policy is not used for all storage operations. Sub-task Resolved Thomas Marqardt Actions
        9.
        regression: Azure tests which capture logs failing with move to SLF4J Sub-task Resolved Steve Loughran Actions
        10.
        wasb throws an exception if you try to create a file and there's no parent directory Sub-task Closed Esfandiar Manii Actions
        11.
        Azure: Concurrent I/O does not work when secure.mode is enabled Sub-task Resolved Thomas Marqardt Actions
        12.
        Azure: Reuse ObjectMapper Sub-task Resolved Mingliang Liu Actions
        13.
        NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code Sub-task Resolved Steve Loughran Actions
        14.
        error message in AzureNativeFileSystemStore.connectUsingAnonymousCredentials has too many spaces Sub-task Resolved Mingliang Liu Actions
        15.
        wasb: add support for caching Authorization and SASKeys Sub-task Resolved Sivaguru Sankaridurg Actions
        16.
        wasb: improve throughput by 34% when account limit exceeded Sub-task Resolved Thomas Marqardt Actions
        17.
        TestWasbRemoteCallHelper failing Sub-task Resolved Esfandiar Manii Actions
        18.
        Honoring sticky bit during Deletion when authorization is enabled in WASB Sub-task Resolved Varada Hemeswari Actions
        19.
        WASB: delete recursive should not fail if a file is deleted Sub-task Resolved Thomas Marqardt Actions
        20.
        PageBlobOutputStream to declare hflush/hsync StreamCapabilities for Sub-task Resolved Unassigned Actions
        21.
        Add support for using container saskeys for all accesses Sub-task Resolved Sivaguru Sankaridurg Actions
        22.
        Azure wasb: getFileStatus not making any auth checks Sub-task Resolved Sivaguru Sankaridurg Actions
        23.
        Restrict Access to setPermission operation when authorization is enabled in WASB Sub-task Resolved Kannapiran Srinivasan Actions
        24.
        Azure: POSIX permissions are taking effect in access() method even when authorization is enabled Sub-task Resolved Santhosh G Nayak Actions
        25.
        Sticky bit implementation for rename() operation in Azure WASB Sub-task Resolved Varada Hemeswari Actions
        26.
        ITestNativeAzureFileSystemLive.testLeaseAsDistributedLock assert failure in parallel run Sub-task Resolved Unassigned Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tmarquardt Thomas Marqardt
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment