VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.6.0
    • 2.7.0
    • fs/s3
    • None

    Description

      Currently s3a buffers files on disk(s) before uploading. This JIRA investigates adding a memory-based upload implementation.

      The motivation is evidently performance: this would be beneficial for users with high network bandwidth to S3 (EC2?) or users that run Hadoop directly on an S3-compatible object store (FYI: my contributions are made in name of Amplidata).

      Attachments

        1. HADOOP-11183-010.patch
          30 kB
          Thomas Demoor
        2. HADOOP-11183-009.patch
          30 kB
          Thomas Demoor
        3. HADOOP-11183-008.patch
          30 kB
          Steve Loughran
        4. HADOOP-11183-007.patch
          29 kB
          Thomas Demoor
        5. HADOOP-11183-006.patch
          28 kB
          Thomas Demoor
        6. HADOOP-11183-005.patch
          26 kB
          Thomas Demoor
        7. HADOOP-11183-004.patch
          22 kB
          Thomas Demoor
        8. design-comments.pdf
          44 kB
          Thomas Demoor
        9. HADOOP-11183.003.patch
          22 kB
          Thomas Demoor
        10. HADOOP-11183.002.patch
          26 kB
          Thomas Demoor
        11. HADOOP-11183.001.patch
          25 kB
          Thomas Demoor

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            thodemoor Thomas Demoor
            thodemoor Thomas Demoor
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment