Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:

      Description

      Redis supports pub/sub out-of-the-box. Flume based event streaming application can make use of Redis publisher sink to publish events to topic(s) in Redis and then subscriber(s) of topic/channel can receive events to do near real-time analytics.

      For reference: http://blog.leftronic.com/leftronic-goes-true-real-time-with-nodejs-and

        Activity

        Hide
        Mubarak Seyed added a comment -

        Initial working patch. Tested in lab environment.
        Review board is down, will submit a code review. Thanks.

        Show
        Mubarak Seyed added a comment - Initial working patch. Tested in lab environment. Review board is down, will submit a code review. Thanks.
        Hide
        Mubarak Seyed added a comment -

        Testing configuration:

        agent.sources=avro-source1
        agent.sinks=redis-sink1
        agent.channels=mem-channel-1
        
        agent.sources.avro-source1.channels=mem-channel-1
        agent.sources.avro-source1.type=avro
        agent.sources.avro-source1.bind=0.0.0.0
        agent.sources.avro-source1.port=9090
        
        agent.channels.mem-channel-1.type = memory
        agent.channels.mem-channel-1.capacity = 1000
        
        agent.sinks.redis-sink1.channel=mem-channel-1
        agent.sinks.redis-sink1.type=org.apache.flume.sink.redis.RedisSink
        agent.sinks.redis-sink1.hostname=0.0.0.0
        agent.sinks.redis-sink1.port=6379
        agent.sinks.redis-sink1.batchsize=1
        agent.sinks.redis-sink1.ttl=300
        agent.sinks.redis-sink1.timeout=300
        agent.sinks.redis-sink1.topic=test_topic
        agent.sinks.redis-sink1.serializer=org.apache.flume.sink.redis.SimpleRedisEventSerializer
        
        Show
        Mubarak Seyed added a comment - Testing configuration: agent.sources=avro-source1 agent.sinks=redis-sink1 agent.channels=mem-channel-1 agent.sources.avro-source1.channels=mem-channel-1 agent.sources.avro-source1.type=avro agent.sources.avro-source1.bind=0.0.0.0 agent.sources.avro-source1.port=9090 agent.channels.mem-channel-1.type = memory agent.channels.mem-channel-1.capacity = 1000 agent.sinks.redis-sink1.channel=mem-channel-1 agent.sinks.redis-sink1.type=org.apache.flume.sink.redis.RedisSink agent.sinks.redis-sink1.hostname=0.0.0.0 agent.sinks.redis-sink1.port=6379 agent.sinks.redis-sink1.batchsize=1 agent.sinks.redis-sink1.ttl=300 agent.sinks.redis-sink1.timeout=300 agent.sinks.redis-sink1.topic=test_topic agent.sinks.redis-sink1.serializer=org.apache.flume.sink.redis.SimpleRedisEventSerializer
        Hide
        Mubarak Seyed added a comment -

        Reviewboard is down at the moment, can someone please review this patch? Thanks.

        Show
        Mubarak Seyed added a comment - Reviewboard is down at the moment, can someone please review this patch? Thanks.
        Hide
        Mubarak Seyed added a comment -

        Posted code review at https://reviews.apache.org/r/5347/

        Show
        Mubarak Seyed added a comment - Posted code review at https://reviews.apache.org/r/5347/

          People

          • Assignee:
            Mubarak Seyed
            Reporter:
            Mubarak Seyed
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development