Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11571 Über-jira: S3a stabilisation phase I
  3. HADOOP-11183

Memory-based S3AOutputstream

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.6.0
    • 2.7.0
    • fs/s3
    • None

    Description

      Currently s3a buffers files on disk(s) before uploading. This JIRA investigates adding a memory-based upload implementation.

      The motivation is evidently performance: this would be beneficial for users with high network bandwidth to S3 (EC2?) or users that run Hadoop directly on an S3-compatible object store (FYI: my contributions are made in name of Amplidata).

      Attachments

        1. HADOOP-11183.001.patch
          25 kB
          Thomas Demoor
        2. HADOOP-11183.002.patch
          26 kB
          Thomas Demoor
        3. HADOOP-11183.003.patch
          22 kB
          Thomas Demoor
        4. design-comments.pdf
          44 kB
          Thomas Demoor
        5. HADOOP-11183-004.patch
          22 kB
          Thomas Demoor
        6. HADOOP-11183-005.patch
          26 kB
          Thomas Demoor
        7. HADOOP-11183-006.patch
          28 kB
          Thomas Demoor
        8. HADOOP-11183-007.patch
          29 kB
          Thomas Demoor
        9. HADOOP-11183-008.patch
          30 kB
          Steve Loughran
        10. HADOOP-11183-009.patch
          30 kB
          Thomas Demoor
        11. HADOOP-11183-010.patch
          30 kB
          Thomas Demoor

        Issue Links

          depends upon

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-10400 Incorporate new S3A FileSystem implementation

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. HADOOP-11463 Replace method-local TransferManager object with S3AFileSystem#transfers

          • Major - Major loss of function.
          • Closed
          is related to

          Sub-task - The sub-task of the issue HADOOP-13560 S3ABlockOutputStream to support huge (many GB) file writes

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              thodemoor Thomas Demoor
              thodemoor Thomas Demoor
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: