Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17566

Über-jira: S3A Hadoop 3.4 features

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Attachments

        Issue Links

        1.
        ITestS3AContractSeek.teardown closes FS before superclass does its cleanup Sub-task Open Unassigned  
        2.
        Add an Audit plugin point for S3A auditing/context Sub-task In Progress Steve Loughran

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 13h 20m
        3.
        S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException Sub-task Open Unassigned  
        4.
        Add a MkdirOperation for chained S3 operations during mkdir Sub-task In Progress Steve Loughran

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        5.
        Magic committer to downgrade abort in cleanup if list uploads fails with access denied Sub-task Open Unassigned  
        6.
        ITestAssumeRole.testAssumeRoleBadInnerAuth failure Sub-task Resolved Steve Loughran

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 50m
        7.
        Use S3 content-range header to update length of an object during reads Sub-task Open Unassigned  
        8.
        S3Guard import can OOM on large imports Sub-task In Progress Steve Loughran

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        9.
        S3A ITestPartialRenamesDeletes.testRenameDirFailsInDelete failure: missing directory marker Sub-task Reopened Steve Loughran  
        10.
        fs.s3a.buffer.dir to be under Yarn container path on yarn applications Sub-task Open Unassigned  
        11.
        whitespace not allowed in paths when saving files to s3a via committer Sub-task Open Unassigned  
        12.
        Support S3 Access Points Sub-task Open Unassigned  
        13.
        Failure of ITestAssumeRole.testRestrictedCommitActions Sub-task Open Steve Loughran  
        14.
        Re-enable optimized copyFromLocal implementation in S3AFileSystem Sub-task Open Unassigned  
        15.
        ITestS3AConfiguration.testProxyConnection failing when s3a bucket probe disabled Sub-task Open Unassigned  
        16.
        S3A AWS Credential provider loading gets confused with isolated classloaders Sub-task In Progress Steve Loughran  
        17.
        S3A (async) ObjectListingIterator to block in hasNext() for results Sub-task Open Steve Loughran

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1.5h
        18.
        S3A deleteObjects hanging/retrying forever Sub-task Open Unassigned  
        19.
        log accepted/rejected fs.s3a.authoritative.path paths @ debug Sub-task Open Unassigned  
        20.
        AWS AssumedRoleCredentialProvider needs ExternalId add Sub-task Open Unassigned  
        21.
        S3A mkdirs to indicate which parent path element refers to a file Sub-task Open Unassigned  
        22.
        transient ITestS3AFileContextStatistics failure -read buffer not filled Sub-task Open Unassigned  
        23.
        S3A DT marshalling to include nested error text in wrapped message Sub-task Open Unassigned  
        24.
        ITestDynamoDBMetadataStore.testTableVersioning failure -DDB deleteItem consistency? Sub-task Open Unassigned  
        25.
        ITestCustomSigner uses absolute paths off the bucket root rather than fork-relative Sub-task Open Unassigned  
        26.
        NPE in s3a byte buffer block upload Sub-task Open Unassigned  
        27.
        GCS to support per-bucket configuration Sub-task Open Unassigned  
        28.
        S3A delegation token binding to support secondary binding list Sub-task In Progress Steve Loughran  
        29.
        S3A client retries on SSL Auth exceptions triggered by "." bucket names Sub-task Open Unassigned  
        30.
        S3AInputStream logging to make it easier to debug file leakage Sub-task Open Unassigned  
        31.
        s3a rename failed during copy, "Unable to copy part" + 200 error code Sub-task Open Unassigned  
        32.
        Filesystem discovery to stop loading implementation classes Sub-task Open Unassigned  
        33.
        Remove transient dependency on hadoop-hdfs-client Sub-task Open Unassigned  
        34.
        S3A input stream to support ByteBufferReadable Sub-task Open Unassigned  
        35.
        Encrypt S3A buffered data on disk Sub-task Open Unassigned  
        36.
        S3a: Failed to reset the request input stream/make S3A uploadPart() retriable Sub-task Open Unassigned  
        37.
        Possible NPE in S3A MultiObjectDeleteSupport error handling Sub-task Open Steve Loughran  
        38.
        Add AWS S3 Transfer acceleration support Sub-task Open Unassigned  
        39.
        Tune hadoop-aws parallel test surefire/failsafe settings Sub-task Open Unassigned  
        40.
        typo in TestNeworkBinding Sub-task Open Steve Loughran  
        41.
        ITestS3AContractSeek teardown closes test FS before superclass can do its cleanup Sub-task Open Unassigned  
        42.
        S3AFileSystem copyFile to propagate etag/version from getObjectMetadata to copy request Sub-task Open Unassigned  
        43.
        Distcp to set S3 Storage Class Sub-task Open Unassigned

        0%

        Original Estimate - 168h
        Remaining Estimate - 168h
        44.
        S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with the throttle backoff policy Sub-task Open Unassigned  
        45.
        ITestS3AContractGetFileStatusV1List may have consistency issues Sub-task Open Unassigned  
        46.
        S3A to support Requester Pays Buckets Sub-task Patch Available Mandus Momberg

        0%

        Original Estimate - 2h
        Remaining Estimate - 2h
        47.
        ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled Sub-task Open Unassigned  
        48.
        S3A DT support to warn when loading expired token Sub-task Open Steve Loughran  
        49.
        S3A can support short user-friendly aliases for configuration of credential providers. Sub-task Open Unassigned  
        50.
        S3AFileStatus to add a serialVersionUID; review & test serialization Sub-task Open Unassigned  
        51.
        S3A to support configuring various AWS S3 client extended options Sub-task Open Unassigned  
        52.
        Report problems w/ local S3A buffer directory meaningfully Sub-task Open Unassigned  

          Activity

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 170h Original Estimate - 170h
                170h
                Remaining:
                Time Spent - 18h 20m Remaining Estimate - 170h
                170h
                Logged:
                Time Spent - 18h 20m Remaining Estimate - 170h
                18h 20m