Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: new service
    • Labels:
      None

      Description

      Add a new service for Flume.

        Issue Links

          Activity

          Hide
          Russell Jurney added a comment -

          I agree with Andrei. A whirr to use flume to log events to s3 in avro would be useful to many people.

          Show
          Russell Jurney added a comment - I agree with Andrei. A whirr to use flume to log events to s3 in avro would be useful to many people.
          Hide
          Andrei Savu added a comment -

          I think this could be really useful if we can also deploy a hdfs / s3 sink.

          http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/

          Show
          Andrei Savu added a comment - I think this could be really useful if we can also deploy a hdfs / s3 sink. http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/
          Hide
          Tom White added a comment -

          Here's a first attempt at this. I can start a flume master and node using the following config:

          whirr.cluster-name=flume
          whirr.instance-templates=1 flume-master,1 flume-node
          whirr.provider=ec2
          whirr.identity=${env:AWS_ACCESS_KEY_ID}
          whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
          whirr.client-cidrs=0.0.0.0/0
          

          This still needs an integration test. Also, the configuration needs some work - I haven't looked at it.

          Show
          Tom White added a comment - Here's a first attempt at this. I can start a flume master and node using the following config: whirr.cluster-name=flume whirr.instance-templates=1 flume-master,1 flume-node whirr.provider=ec2 whirr.identity=${env:AWS_ACCESS_KEY_ID} whirr.credential=${env:AWS_SECRET_ACCESS_KEY} whirr.client-cidrs=0.0.0.0/0 This still needs an integration test. Also, the configuration needs some work - I haven't looked at it.

            People

            • Assignee:
              Unassigned
              Reporter:
              Tom White
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development