Details
-
Question
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Flume kafka source has been used with spillable memory channel to dump messages into kite dataset.
Every message from a kafka topic is around 180 bytes.
And messages/sec is approx 80k/sec.
So, flume spill-able memory channel has been configured with,
memory capacity : 5000000
transaction capacity : 50000
batch size : 50000
byteCapacityBufferPercentage : 30
Upon addition of parameter byteCapacity with value 900000000 (memoryCapacity * size of each message/event i.e 5000000*180),
We see the kafka bytes out/sec due to flume kafka source is very less compared to kafka bytes in/sec.
If byteCapacity is removed from flume config, the bytesOut/sec seems to increase and equal to bytesIn/sec.
What is the purpose of having byteCapacity and what should be the right way to calculate the same ?