Description
KafkaBasedLog subclasses use an infinite retry on producer sends, using a callback. Sometimes, when specific errors are encountered, the callback is invoked in the send call, on the calling thread. If this happens enough times, a stack overflow happens.
Example stacktrace from 2.5 (but the newest code can also encounter the same):
2023-01-14 12:48:23,487 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-1} Task threw an uncaught and unrecoverable exception java.lang.StackOverflowError: null at org.apache.kafka.common.metrics.stats.SampledStat.record(SampledStat.java:50) at org.apache.kafka.common.metrics.stats.Rate.record(Rate.java:60) at org.apache.kafka.common.metrics.stats.Meter.record(Meter.java:80) at org.apache.kafka.common.metrics.Sensor.record(Sensor.java:188) at org.apache.kafka.common.metrics.Sensor.record(Sensor.java:178) at org.apache.kafka.clients.producer.internals.BufferPool.recordWaitTime(BufferPool.java:202) at org.apache.kafka.clients.producer.internals.BufferPool.allocate(BufferPool.java:147) at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:221) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:941) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862) at org.apache.kafka.connect.util.KafkaBasedLog.send(KafkaBasedLog.java:238) at org.apache.kafka.connect.storage.KafkaStatusBackingStore$4.onCompletion(KafkaStatusBackingStore.java:298) ... at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:959) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862) at org.apache.kafka.connect.util.KafkaBasedLog.send(KafkaBasedLog.java:238) at org.apache.kafka.connect.storage.KafkaStatusBackingStore$4.onCompletion(KafkaStatusBackingStore.java:298)
Note the repeated KafkaProducer.send -> KafkaProducer.doSend -> KafkaStatusBackingStore$4.onCompletion calls, causing the issue.
Attachments
Issue Links
- is related to
-
KAFKA-14929
Flaky KafkaStatusBackingStoreFormatTest#putTopicStateRetriableFailure
-
- Resolved
-
- links to