Flume
  1. Flume
  2. FLUME-1180

ChannelSelectorFactory creates incorrect selector for multiplexing selector type

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.2.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      loadSources() of PropertiesFileConfigurationProvider creates a config with "." for key values and ChannelSelectorFactory tries to get value for key "type", which defaults to ReplicatingChannelSelector as "type" will not be found in config.

      
      agent.sources = avro-source1
      agent.sinks = hdfs-sink1 hdfs-sink2
      agent.channels = mem-channel-1 mem-channel-2
      
      #source
      agent.sources.avro-source1.channels = mem-channel-1 mem-channel-2
      agent.sources.avro-source1.type = avro
      agent.sources.avro-source1.bind = 0.0.0.0
      agent.sources.avro-source1.port = 9090
      agent.sources.avro-source1.selector.type = multiplexing
      agent.sources.avro-source1.selector.header = tableName
      agent.sources.avro-source1.selector.mapping.table1 = mem-channel-1
      agent.sources.avro-source1.selector.mapping.table2 = mem-channel-2
      agent.sources.avro-source1.selector.default = mem-channel-1
      
      #sinks
      agent.sinks.hdfs-sink1.channel = mem-channel-1
      agent.sinks.hdfs-sink2.channel = mem-channel-2
      
      agent.sinks.hdfs-sink1.type = hdfs
      agent.sinks.hdfs-sink1.hdfs.batchSize = 1000
      agent.sinks.hdfs-sink1.hdfs.txnEventMax = 1000
      agent.sinks.hdfs-sink1.hdfs.fileType = DataStream
      agent.sinks.hdfs-sink1.hdfs.rollSize = 1073741824
      agent.sinks.hdfs-sink1.hdfs.rollCount = 0
      agent.sinks.hdfs-sink1.hdfs.rollInterval = 0
      agent.sinks.hdfs-sink1.serializer = com.mycompany.Table1AvroEventSerializer$Builder
      agent.sinks.hdfs-sink1.serializer.compressionCodec = snappy
      agent.sinks.hdfs-sink1.serializer.syncIntervalBytes = 2048000
      agent.sinks.hdfs-sink1.hdfs.path = hdfs://NN/logs/table1/%m%d%Y/%H
      
      agent.sinks.hdfs-sink2.type = hdfs
      agent.sinks.hdfs-sink2.hdfs.batchSize = 1000
      agent.sinks.hdfs-sink2.hdfs.txnEventMax = 1000
      agent.sinks.hdfs-sink2.hdfs.fileType = DataStream
      agent.sinks.hdfs-sink2.hdfs.rollSize = 1073741824
      agent.sinks.hdfs-sink2.hdfs.rollCount = 0
      agent.sinks.hdfs-sink2.hdfs.rollInterval = 300
      agent.sinks.hdfs-sink2.serializer = com.mycompany.Table2AvroEventSerializer$Builder
      agent.sinks.hdfs-sink2.serializer.compressionCodec = snappy
      agent.sinks.hdfs-sink2.serializer.syncIntervalBytes = 2048000
      agent.sinks.hdfs-sink2.hdfs.path = hdfs://NN/logs/table2/%m%d%Y/%H
      
      agent.channels.mem-channel-1.type = memory
      agent.channels.mem-channel-1.capacity = 100000
      agent.channels.mem-channel-1.transactionCapacity = 1000
      
      agent.channels.mem-channel-2.type = memory
      agent.channels.mem-channel-2.capacity = 100000
      agent.channels.mem-channel-2.transactionCapacity = 1000
      

      but the config for selector gets parsed as

      {.type=multiplexing, .default=mem-channel-1, .mapping.table2=mem-channel-2, .mapping.table1=mem-channel-1, .header=tableName}
      

      The fix is in loadSources() of PropertiesFileConfigurationProvider

      Map<String, String> selectorConfig = context.getSubProperties("selector" + "."); //fix
      

        Activity

        Hide
        Mubarak Seyed added a comment -

        The attached file is a patch. Unit tested and tested in lab environment. Thanks.

        Show
        Mubarak Seyed added a comment - The attached file is a patch. Unit tested and tested in lab environment. Thanks.
        Hide
        Hari Shreedharan added a comment -

        Looks good!

        Show
        Hari Shreedharan added a comment - Looks good!
        Hide
        Arvind Prabhakar added a comment -

        +1.

        Thanks for the patch Mubarak.

        Show
        Arvind Prabhakar added a comment - +1. Thanks for the patch Mubarak.
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Mubarak!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Mubarak!
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #195 (See https://builds.apache.org/job/flume-trunk/195/)
        FLUME-1180. ChannelSelectorFactory creates incorrect selector for multiplexing selector type.

        (Mubarak Seyed via Arvind Prabhakar) (Revision 1334468)

        Result = SUCCESS
        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1334468
        Files :

        • /incubator/flume/trunk/flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
        Show
        Hudson added a comment - Integrated in flume-trunk #195 (See https://builds.apache.org/job/flume-trunk/195/ ) FLUME-1180 . ChannelSelectorFactory creates incorrect selector for multiplexing selector type. (Mubarak Seyed via Arvind Prabhakar) (Revision 1334468) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1334468 Files : /incubator/flume/trunk/flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java

          People

          • Assignee:
            Mubarak Seyed
            Reporter:
            Mubarak Seyed
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development