[FLUME-1180] ChannelSelectorFactory creates incorrect selector for multiplexing selector type

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Sinks+Sources
• Labels: newbie
• Assignee: Mubarak Seyed
• Reporter: Mubarak Seyed
• Created: 05/May/12 01:56
• Updated: 05/May/12 20:12
• Resolved: 05/May/12 19:21

Description

loadSources() of PropertiesFileConfigurationProvider creates a config with "." for key values and ChannelSelectorFactory tries to get value for key "type", which defaults to ReplicatingChannelSelector as "type" will not be found in config.

agent.sources = avro-source1
agent.sinks = hdfs-sink1 hdfs-sink2
agent.channels = mem-channel-1 mem-channel-2

#source
agent.sources.avro-source1.channels = mem-channel-1 mem-channel-2
agent.sources.avro-source1.type = avro
agent.sources.avro-source1.bind = 0.0.0.0
agent.sources.avro-source1.port = 9090
agent.sources.avro-source1.selector.type = multiplexing
agent.sources.avro-source1.selector.header = tableName
agent.sources.avro-source1.selector.mapping.table1 = mem-channel-1
agent.sources.avro-source1.selector.mapping.table2 = mem-channel-2
agent.sources.avro-source1.selector.default = mem-channel-1

#sinks
agent.sinks.hdfs-sink1.channel = mem-channel-1
agent.sinks.hdfs-sink2.channel = mem-channel-2

agent.sinks.hdfs-sink1.type = hdfs
agent.sinks.hdfs-sink1.hdfs.batchSize = 1000
agent.sinks.hdfs-sink1.hdfs.txnEventMax = 1000
agent.sinks.hdfs-sink1.hdfs.fileType = DataStream
agent.sinks.hdfs-sink1.hdfs.rollSize = 1073741824
agent.sinks.hdfs-sink1.hdfs.rollCount = 0
agent.sinks.hdfs-sink1.hdfs.rollInterval = 0
agent.sinks.hdfs-sink1.serializer = com.mycompany.Table1AvroEventSerializer$Builder
agent.sinks.hdfs-sink1.serializer.compressionCodec = snappy
agent.sinks.hdfs-sink1.serializer.syncIntervalBytes = 2048000
agent.sinks.hdfs-sink1.hdfs.path = hdfs://NN/logs/table1/%m%d%Y/%H

agent.sinks.hdfs-sink2.type = hdfs
agent.sinks.hdfs-sink2.hdfs.batchSize = 1000
agent.sinks.hdfs-sink2.hdfs.txnEventMax = 1000
agent.sinks.hdfs-sink2.hdfs.fileType = DataStream
agent.sinks.hdfs-sink2.hdfs.rollSize = 1073741824
agent.sinks.hdfs-sink2.hdfs.rollCount = 0
agent.sinks.hdfs-sink2.hdfs.rollInterval = 300
agent.sinks.hdfs-sink2.serializer = com.mycompany.Table2AvroEventSerializer$Builder
agent.sinks.hdfs-sink2.serializer.compressionCodec = snappy
agent.sinks.hdfs-sink2.serializer.syncIntervalBytes = 2048000
agent.sinks.hdfs-sink2.hdfs.path = hdfs://NN/logs/table2/%m%d%Y/%H

agent.channels.mem-channel-1.type = memory
agent.channels.mem-channel-1.capacity = 100000
agent.channels.mem-channel-1.transactionCapacity = 1000

agent.channels.mem-channel-2.type = memory
agent.channels.mem-channel-2.capacity = 100000
agent.channels.mem-channel-2.transactionCapacity = 1000

but the config for selector gets parsed as

{.type=multiplexing, .default=mem-channel-1, .mapping.table2=mem-channel-2, .mapping.table1=mem-channel-1, .header=tableName}

The fix is in loadSources() of PropertiesFileConfigurationProvider

Map<String, String> selectorConfig = context.getSubProperties("selector" + "."); //fix