[KAFKA-7625] Kafka Broker node JVM crash - kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Bug
• Affects Version/s: 2.0.0
• Component/s: core

Environment

 environment:os.version=2.6.32-754.2.1.el6.x86_64
java.version=1.8.0_92
environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
Kafka commitId : 3402a8361b734732

• Assignee: Unassigned
• Reporter: Sebastian Puzoń
• Created: 14/Nov/18 16:24
• Updated: 23/Sep/19 12:08
• Resolved: 27/Dec/18 15:59

Description

I observe broker node JVM crashes with same problematic frame:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff4a2588261, pid=24681, tid=0x00007ff3b9bb1700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 9736 C1 kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Either; (518 bytes) @ 0x00007ff4a2588261 [0x00007ff4a25871a0+0x10c1]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007ff4b356f800):  JavaThread "kafka-request-handler-3" daemon [_thread_in_Java, id=24781, stack(0x00007ff3b9ab1000,0x00007ff3b9bb2000)]

Stack: [0x00007ff3b9ab1000,0x00007ff3b9bb2000],  sp=0x00007ff3b9bafca0,  free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 9736 C1 kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Either; (518 bytes) @ 0x00007ff4a2588261 [0x00007ff4a25871a0+0x10c1]
J 10456 C2 kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$5(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;ILscala/Option;)Lscala/util/Either; (192 bytes) @ 0x00007ff4a1d413f0 [0x00007ff4a1d41240+0x1b0]
J 9303 C1 kafka.coordinator.transaction.TransactionCoordinator$$Lambda$1107.apply(Ljava/lang/Object;)Ljava/lang/Object; (32 bytes) @ 0x00007ff4a245f55c [0x00007ff4a245f3c0+0x19c]
J 10018 C2 scala.util.Either$RightProjection.flatMap(Lscala/Function1;)Lscala/util/Either; (43 bytes) @ 0x00007ff4a1f242c4 [0x00007ff4a1f24260+0x64]
J 9644 C1 kafka.coordinator.transaction.TransactionCoordinator.sendTxnMarkersCallback$1(Lorg/apache/kafka/common/protocol/Errors;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lscala/Function1;ILkafka/coordinator/transaction/TxnTransitMetadata;)V (251 bytes) @ 0x00007ff4a1ef6254 [0x00007ff4a1ef5120+0x1134]
J 9302 C1 kafka.coordinator.transaction.TransactionCoordinator$$Lambda$1106.apply(Ljava/lang/Object;)Ljava/lang/Object; (40 bytes) @ 0x00007ff4a24747ec [0x00007ff4a24745a0+0x24c]
J 10125 C2 kafka.coordinator.transaction.TransactionStateManager.updateCacheCallback$1(Lscala/collection/Map;Ljava/lang/String;ILkafka/coordinator/transaction/TxnTransitMetadata;Lscala/Function1;Lscala/Function1;Lorg/apache/kafka/common/TopicPartition;)V (892 bytes) @ 0x00007ff4a27045ec [0x00007ff4a2703c60+0x98c]
J 10051 C2 kafka.coordinator.transaction.TransactionStateManager$$Lambda$814.apply(Ljava/lang/Object;)Ljava/lang/Object; (36 bytes) @ 0x00007ff4a1a9cd08 [0x00007ff4a1a9cc80+0x88]
J 9349 C2 kafka.server.DelayedProduce.tryComplete()Z (52 bytes) @ 0x00007ff4a1e46e5c [0x00007ff4a1e46980+0x4dc]
J 10111 C2 kafka.server.DelayedOperationPurgatory.tryCompleteElseWatch(Lkafka/server/DelayedOperation;Lscala/collection/Seq;)Z (147 bytes) @ 0x00007ff4a1c6e000 [0x00007ff4a1c6df20+0xe0]
J 10448 C2 kafka.server.ReplicaManager.appendRecords(JSZZLscala/collection/Map;Lscala/Function1;Lscala/Option;Lscala/Function1;)V (237 bytes) @ 0x00007ff4a2340b6c [0x00007ff4a233f3e0+0x178c]
J 10050 C2 kafka.coordinator.transaction.TransactionStateManager.$anonfun$appendTransactionToLog$13(Lkafka/coordinator/transaction/TransactionStateManager;Ljava/lang/String;ILkafka/coordinator/transaction/TxnTransitMetadata;Lscala/Function1;Lscala/Function1;Lorg/apache/kafka/common/TopicPartition;Lscala/collection/immutable/Map;)V (294 bytes) @ 0x00007ff4a25a2384 [0x00007ff4a25a1800+0xb84]
J 10112 C2 kafka.coordinator.transaction.TransactionStateManager$$Lambda$812.apply$mcV$sp()V (36 bytes) @ 0x00007ff4a1b24e64 [0x00007ff4a1b24de0+0x84]
J 4800 C2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007ff4a1ad73dc [0x00007ff4a1ad73a0+0x3c]
J 6365 C2 kafka.utils.CoreUtils$.inReadLock(Ljava/util/concurrent/locks/ReadWriteLock;Lscala/Function0;)Ljava/lang/Object; (12 bytes) @ 0x00007ff4a13c7794 [0x00007ff4a13c7720+0x74]
J 10037 C2 kafka.coordinator.transaction.TransactionStateManager.appendTransactionToLog(Ljava/lang/String;ILkafka/coordinator/transaction/TxnTransitMetadata;Lscala/Function1;Lscala/Function1;)V (152 bytes) @ 0x00007ff4a26cd90c [0x00007ff4a26c9f40+0x39cc]
J 10447 C2 kafka.server.KafkaApis.handleEndTxnRequest(Lkafka/network/RequestChannel$Request;)V (118 bytes) @ 0x00007ff4a1fbb154 [0x00007ff4a1fb9fa0+0x11b4]
J 6316 C2 kafka.server.KafkaApis.handle(Lkafka/network/RequestChannel$Request;)V (1154 bytes) @ 0x00007ff4a19538c0 [0x00007ff4a1953120+0x7a0]
J 8822% C2 kafka.server.KafkaRequestHandler.run()V (258 bytes) @ 0x00007ff4a1d43bc4 [0x00007ff4a1d43a60+0x164]
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

 

 Attached crash reports:

 

/apps/solr/kafka/kafka_2.12-2.0.0 $ ls -lth hs_err_pid*
-rwxrwxrwx 1 res2 res 424K Nov 14 14:02 hs_err_pid24681.log
-rwxrwxrwx 1 res2 res 427K Nov 13 05:18 hs_err_pid4194.log
-rwxrwxrwx 1 res2 res 423K Nov 12 17:06 hs_err_pid15119.log
-rwxrwxrwx 1 res2 res 425K Nov 12 04:56 hs_err_pid4299.log
-rwxrwxrwx 1 res2 res 423K Nov 11 16:32 hs_err_pid25701.log
-rwxrwxrwx 1 res2 res 424K Nov 11 03:44 hs_err_pid22373.log
-rwxrwxrwx 1 res2 res 426K Nov 10 15:24 hs_err_pid25513.log
-rwxrwxrwx 1 res2 res 421K Nov 10 02:16 hs_err_pid10238.log
-rwxrwxrwx 1 res2 res 435K Nov  8 11:45 hs_err_pid22386.log
-rwxrwxrwx 1 res2 res 417K Nov  7 20:38 hs_err_pid26844.log
-rwxrwxrwx 1 res2 res 419K Nov  7 08:04 hs_err_pid27290.log
-rwxrwxrwx 1 res2 res 419K Nov  6 17:37 hs_err_pid19131.log
-rwxrwxrwx 1 res2 res 418K Nov  6 03:50 hs_err_pid22633.log