Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
0.8.2.1
-
None
-
None
-
None
Description
I've been playing around with Mirror Maker (version from trunk, dated July 7th) and got a few problems, most noticeable of which is that MirrorMaker exhausts it's memory pool, even though it's size set to relatively huge value of 132 MB, and individual messages are around 2 KB. Batch size is set to just 2 messages (full configs are attached).
[2015-07-21 15:19:52,915] FATAL [mirrormaker-thread-1] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread) org.apache.kafka.clients.producer.BufferExhaustedException: You have exhausted the 134217728 bytes of memory you configured for the client and the client is configured to error rather than block when memory is exhausted. at org.apache.kafka.clients.producer.internals.BufferPool.allocate(BufferPool.java:124) at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:172) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:388) at kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:380) at kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:311) at kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:311) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:311)
Am I doing something wrong? Any help in further diagnosing of this problem might be handy.