Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2069

Issue with Flume load balancing round robin

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 1.3.1
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      I am not sure if this is a bug. I have an http to hdfs scenario, 3 channels with 3 hdfs sinks, I have configured it to do load balancing with round robin, but when the request arrive it gets replicated into all the sinks instead of hitting the sink with round robin order, so I end up with the same data replicated by all sinks.

      here are my configs, I have not included the source configs here

      a1.sinks.k1.type = hdfs
      a1.sinks.k1.hdfs.path = /tmp/hadoop-oawad/dfs/name2/%y-%m-%d/%H%M/%S
      a1.sinks.k1.hdfs.fileType = DataStream
      a1.sinks.k1.hdfs.writeFormat = Text
      a1.sinks.k1.hdfs.filePrefix = events
      a1.sinks.k1.hdfs.batchSize = 1000

      a1.sinks.k2.type = hdfs
      a1.sinks.k2.hdfs.path = /tmp/hadoop-oawad/dfs/name2/%y-%m-%d/%H%M/%S
      a1.sinks.k2.hdfs.fileType = DataStream
      a1.sinks.k2.hdfs.writeFormat = Text
      a1.sinks.k2.hdfs.filePrefix = events
      a1.sinks.k2.hdfs.batchSize = 1000

      a1.sinks.k3.type = hdfs
      a1.sinks.k3.hdfs.path = /tmp/hadoop-oawad/dfs/name2/%y-%m-%d/%H%M/%S
      a1.sinks.k3.hdfs.fileType = DataStream
      a1.sinks.k3.hdfs.writeFormat = Text
      a1.sinks.k3.hdfs.filePrefix = events
      a1.sinks.k3.hdfs.batchSize = 1000

      a1.sinkgroups = g1
      a1.sinkgroups.g1.sinks = k1 k2 k3
      a1.sinkgroups.g1.processor.type = load_balance
      a1.sinkgroups.g1.processor.selector = round_robin

      am I missing something here?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              oawad79 Osama Awad
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: