Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Currently s3a buffers files on disk(s) before uploading. This JIRA investigates adding a memory-based upload implementation.

      The motivation is evidently performance: this would be beneficial for users with high network bandwidth to S3 (EC2?) or users that run Hadoop directly on an S3-compatible object store (FYI: my contributions are made in name of Amplidata).

        Attachments

        1. design-comments.pdf
          44 kB
          Thomas Demoor
        2. HADOOP-11183.001.patch
          25 kB
          Thomas Demoor
        3. HADOOP-11183.002.patch
          26 kB
          Thomas Demoor
        4. HADOOP-11183.003.patch
          22 kB
          Thomas Demoor
        5. HADOOP-11183-004.patch
          22 kB
          Thomas Demoor
        6. HADOOP-11183-005.patch
          26 kB
          Thomas Demoor
        7. HADOOP-11183-006.patch
          28 kB
          Thomas Demoor
        8. HADOOP-11183-007.patch
          29 kB
          Thomas Demoor
        9. HADOOP-11183-008.patch
          30 kB
          Steve Loughran
        10. HADOOP-11183-009.patch
          30 kB
          Thomas Demoor
        11. HADOOP-11183-010.patch
          30 kB
          Thomas Demoor

          Issue Links

            Activity

              People

              • Assignee:
                thodemoor Thomas Demoor
                Reporter:
                thodemoor Thomas Demoor
              • Votes:
                1 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: