Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-951

Implement a direct S3 Sink for Flume

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.0.0
    • None
    • None
    • None

    Attachments

      1. flume-s3-sink.tar.gz
        5 kB
        Michael Katzenellenbogen

      Activity

        Assigning this to Michael K as he has already done some work in this regard.

        aprabhakar Arvind Prabhakar added a comment - Assigning this to Michael K as he has already done some work in this regard.

        FLUME-1015 was created too, but I closed that. Eric mentioned to use HDFS abstraction for S3.

        alo.alt Alexander Alten-Lorenz added a comment - FLUME-1015 was created too, but I closed that. Eric mentioned to use HDFS abstraction for S3.

        This Jira is for implementing a direct S3 sink instead of S3 over HDFS.

        aprabhakar Arvind Prabhakar added a comment - This Jira is for implementing a direct S3 sink instead of S3 over HDFS.
        brocknoland Brock Noland added a comment - also there is this https://github.com/Hobsons/hobsons-flumeng-s3-sink
        brocknoland Brock Noland added a comment -

        Hi Michael,

        Since you haven't worked on this lately, do you mind if I take it up?

        brocknoland Brock Noland added a comment - Hi Michael, Since you haven't worked on this lately, do you mind if I take it up?

        Sure, attached is the sink I wrote, please feel free to use that (ASF license/copyright granted).

        michaelk Michael Katzenellenbogen added a comment - Sure, attached is the sink I wrote, please feel free to use that (ASF license/copyright granted).
        nitin_matrix Nitin Verma added a comment -

        Michael,

        S3BufferedWriter(s) are to get over the inability of s3 to append, I do understand that. Considering sinks write to into storage those have inability to append.

        • Framework should introduce a semantic like this.
        • Channels only buffer, we should maintain that.
        • Channels may start supporting minimum batch size. Alternate to buffered writers in sinks.

        Regards,

        Nitin Verma

        nitin_matrix Nitin Verma added a comment - Michael, S3BufferedWriter(s) are to get over the inability of s3 to append, I do understand that. Considering sinks write to into storage those have inability to append. Framework should introduce a semantic like this. Channels only buffer, we should maintain that. Channels may start supporting minimum batch size. Alternate to buffered writers in sinks. Regards, Nitin Verma

        People

          brocknoland Brock Noland
          aprabhakar Arvind Prabhakar
          Votes:
          2 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

            Created:
            Updated: