[KAFKA-5154] Kafka Streams throws NPE during rebalance

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.2.0
• Fix Version/s: 0.11.0.0
• Component/s: streams
• Assignee: Damian Guy
• Reporter: Lukas Gemela
• Created: 02/May/17 07:57
• Updated: 16/Feb/18 20:39
• Resolved: 01/Aug/17 16:03

Description

please see attached log, Kafka streams throws NullPointerException during rebalance, which is caught by our custom exception handler

2017-04-30T17:44:17,675 INFO  kafka-coordinator-heartbeat-thread | hades org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead() @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) dead for group hades
2017-04-30T17:44:27,395 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) for group hades.
2017-04-30T17:44:27,941 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare() @393 - Revoking previously assigned partitions [poseidonIncidentFeed-27, poseidonIncidentFeed-29, poseidonIncidentFeed-30, poseidonIncidentFeed-18] for group hades
2017-04-30T17:44:27,947 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest() @407 - (Re-)joining group hades
2017-04-30T17:44:48,468 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest() @407 - (Re-)joining group hades
2017-04-30T17:44:53,628 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest() @407 - (Re-)joining group hades
2017-04-30T17:45:09,587 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest() @407 - (Re-)joining group hades
2017-04-30T17:45:11,961 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() @375 - Successfully joined group hades with generation 99
2017-04-30T17:45:13,126 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete() @252 - Setting newly assigned partitions [poseidonIncidentFeed-11, poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
2017-04-30T17:46:37,254 INFO  kafka-coordinator-heartbeat-thread | hades org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead() @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) dead for group hades
2017-04-30T18:04:25,993 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) for group hades.
2017-04-30T18:04:29,401 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare() @393 - Revoking previously assigned partitions [poseidonIncidentFeed-11, poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
2017-04-30T18:05:10,877 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest() @407 - (Re-)joining group hades
2017-05-01T00:01:55,707 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead() @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) dead for group hades
2017-05-01T00:01:59,027 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) for group hades.
2017-05-01T00:01:59,031 ERROR StreamThread-1 org.apache.kafka.streams.processor.internals.StreamThread.run() @376 - stream-thread [StreamThread-1] Streams application error during processing:
 java.lang.NullPointerException
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:619) ~[kafka-streams-0.10.2.0.jar!/:?]
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368) [kafka-streams-0.10.2.0.jar!/:?]
2017-05-01T00:02:00,038 INFO  StreamThread-1 org.apache.kafka.clients.producer.KafkaProducer.close() @689 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2017-05-01T00:02:00,949 WARN  StreamThread-1 org.apache.kafka.streams.processor.internals.StreamThread.setState() @160 - Unexpected state transition from PARTITIONS_REVOKED to NOT_RUNNING
2017-05-01T00:02:00,951 ERROR StreamThread-1 com.williamhill.trading.platform.hades.kafka.KafkaStreamManager.uncaughtException() @104 - UncaughtException in thread StreamThread-1, stopping kafka streams
 java.lang.NullPointerException
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:619) ~[kafka-streams-0.10.2.0.jar!/:?]
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368) ~[kafka-streams-0.10.2.0.jar!/:?]
2017-05-01T00:02:01,076 WARN  kafka-streams-close-thread org.apache.kafka.streams.processor.internals.StreamThread.setState() @160 - Unexpected state transition from NOT_RUNNING to PENDING_SHUTDOWN

Issue Links

• incorporates: KAFKA-5430 new consumers getting data for revoked partitions
• incorporates: KAFKA-6306 Auto-commit of offsets fail, and not recover forever...
• links to: GitHub Pull Request #3181