Flume
  1. Flume
  2. FLUME-951

Implement a direct S3 Sink for Flume

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    1. flume-s3-sink.tar.gz
      5 kB
      Michael Katzenellenbogen

      Activity

      Hide
      Nitin Verma added a comment -

      Michael,

      S3BufferedWriter(s) are to get over the inability of s3 to append, I do understand that. Considering sinks write to into storage those have inability to append.

      • Framework should introduce a semantic like this.
      • Channels only buffer, we should maintain that.
      • Channels may start supporting minimum batch size. Alternate to buffered writers in sinks.

      Regards,

      Nitin Verma

      Show
      Nitin Verma added a comment - Michael, S3BufferedWriter(s) are to get over the inability of s3 to append, I do understand that. Considering sinks write to into storage those have inability to append. Framework should introduce a semantic like this. Channels only buffer, we should maintain that. Channels may start supporting minimum batch size. Alternate to buffered writers in sinks. Regards, Nitin Verma
      Hide
      Michael Katzenellenbogen added a comment -

      Sure, attached is the sink I wrote, please feel free to use that (ASF license/copyright granted).

      Show
      Michael Katzenellenbogen added a comment - Sure, attached is the sink I wrote, please feel free to use that (ASF license/copyright granted).
      Hide
      Brock Noland added a comment -

      Hi Michael,

      Since you haven't worked on this lately, do you mind if I take it up?

      Show
      Brock Noland added a comment - Hi Michael, Since you haven't worked on this lately, do you mind if I take it up?
      Hide
      Brock Noland added a comment -
      Show
      Brock Noland added a comment - also there is this https://github.com/Hobsons/hobsons-flumeng-s3-sink
      Hide
      Arvind Prabhakar added a comment -

      This Jira is for implementing a direct S3 sink instead of S3 over HDFS.

      Show
      Arvind Prabhakar added a comment - This Jira is for implementing a direct S3 sink instead of S3 over HDFS.
      Hide
      Alexander Alten-Lorenz added a comment -

      FLUME-1015 was created too, but I closed that. Eric mentioned to use HDFS abstraction for S3.

      Show
      Alexander Alten-Lorenz added a comment - FLUME-1015 was created too, but I closed that. Eric mentioned to use HDFS abstraction for S3.
      Hide
      Arvind Prabhakar added a comment -

      Assigning this to Michael K as he has already done some work in this regard.

      Show
      Arvind Prabhakar added a comment - Assigning this to Michael K as he has already done some work in this regard.

        People

        • Assignee:
          Brock Noland
          Reporter:
          Arvind Prabhakar
        • Votes:
          5 Vote for this issue
          Watchers:
          6 Start watching this issue

          Dates

          • Created:
            Updated:

            Development