Blocker Description
I am using Kafka High Level Consumer, version kafka_2.8.0 - 0.8.1.1 with zkclient - 0.3 and I get OOME just after 10/15 minutes, My volume test setup has just one topic with 10 partitions with continuous message (size ~500KB) flow and below are my configuration;
zookeeper.connect=localhost:2181,localhost:2181
group.id=tc
consumer.id=tc
zookeeper.sync.time.ms=200
zookeeper.connection.timeout.ms=10000
zookeeper.session.timeout.ms=10000
fetch.size=500000
fetch.message.max.bytes=1000000
auto.commit.enable=true
auto.commit.interval.ms=100
auto.offset.reset=largest
queued.max.message.chunks=1
backoff.increment.ms=1000
rebalance.max.retries=10
rebalance.retries.max=10
rebalance.backoff.ms=10000
refresh.leader.backoff.ms=20000
consumer.timeout.ms=50000
Memory histogram shows 60% of memory consumed by byte[] and most of the remaining by char[] and HashMap$Node. In my various tries to recover from the situation. I observed "metric-*" thread live even after I shutdown Kafka connector?