[FLUME-3213] TransactionCapacity for spillableMemoryChannel with overflowCapacity set to '0' is limited to 1000000.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0

Environment

Below is the setup info,

Channel :SPILLABLEMEMORY

byteCapacityBufferPercentage = 10

transactionCapacity = 2000000

memoryCapacity = 17000000

overflowCapacity = 0

Kafka source and kite data set sink is being used.

• Assignee: Unassigned
• Reporter: kaushik srinivas
• Created: 17/Jan/18 07:07
• Updated: 17/Jan/18 07:07

Description

Intent is to make this spillabeMemory channel work as pure memory channel.

So the parameter,

overflowCapacity is set with value 0.

 

But then observing exceptions and by default capacity is being set to 1000000.

Error 1:

2018-01-17 05:57:26,783 (lifecycleSupervisor-1-0-EventThread) [WARN - org.apache.flume.channel.file.FileChannel.configure(FileChannel.java:167)] Invalid capacity specified, initializing channel to default capacity of 1000000

2018-01-17 05:57:26,787 (lifecycleSupervisor-1-0-EventThread) [ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Channel channel_store_sales has been removed due to an error during configuration

 

Impact of this is on the transaction capacity.

Error1 sets capacity to 10,00,000 and thus transactionCapacity is not allowed to be set more than the capacity value of 1000000 limit.

 

Error2:

java.lang.IllegalStateException: File Channel transaction capacity cannot be greater than the capacity of the channel.

at com.google.common.base.Preconditions.checkState(Preconditions.java:145)

at org.apache.flume.channel.file.FileChannel.configure(FileChannel.java:186)

at org.apache.flume.channel.SpillableMemoryChannel.configure(SpillableMemoryChannel.java:745)

at org.apache.flume.conf.Configurables.configure(Configurables.java:41)

at org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:199)

at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:100)

at org.apache.flume.node.PollingZooKeeperConfigurationProvider.refreshConfiguration(PollingZooKeeperConfigurationProvider.java:104)

at org.apache.flume.node.PollingZooKeeperConfigurationProvider.access$000(PollingZooKeeperConfigurationProvider.java:36)

at org.apache.flume.node.PollingZooKeeperConfigurationProvider$1.nodeChanged(PollingZooKeeperConfigurationProvider.java:78)

at org.apache.curator.framework.recipes.cache.NodeCache$4.apply(NodeCache.java:293)

at org.apache.curator.framework.recipes.cache.NodeCache$4.apply(NodeCache.java:287)

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)

at org.apache.curator.framework.recipes.cache.NodeCache.setNewData(NodeCache.java:284)

at org.apache.curator.framework.recipes.cache.NodeCache.processBackgroundResult(NodeCache.java:252)

at org.apache.curator.framework.recipes.cache.NodeCache.access$300(NodeCache.java:53)

at org.apache.curator.framework.recipes.cache.NodeCache$3.processResult(NodeCache.java:111)

at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:715)

at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:502)

at org.apache.curator.framework.imps.GetDataBuilderImpl$3.processResult(GetDataBuilderImpl.java:254)

at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:561)

 

So due to this, transactionCapacity for spillableMemoryChannel with overflowCapacity set to 0 is limited to 1000000.

So overflowCapacity has to be set with a high number in order to increase the transactionCapacity beyond1000000, which should not be the ideal scenario.