[KAFKA-6844] Race condition between StreamThread and GlobalStreamThread stopping

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.0.1, 1.1.0
• Fix Version/s: 2.0.0
• Component/s: streams
• Assignee: Bill Bejeck
• Reporter: Bill Bejeck
• Created: 01/May/18 22:26
• Updated: 04/May/18 22:18
• Resolved: 04/May/18 22:18

Description

There can be a race condition where shut down is called on a StreamThread then shut down is called on a GlobalStreamThread, but the StreamThread can be delayed in shutting down,  and the GlobalStreamThread can shutdown first.

 

If the StreamThread tries to access a GlobalStateStore before closing the user can get an exception stating  "..Store xxx is currently closed "

Here's a redacted partial log file showing this process:

2018-04-23 12:54:10 [INFO] [logger] DataExportTopology:86 - Closing streams
2018-04-23 12:54:10 [INFO] [logger] KafkaStreams:346 - stream-client [redacted-info] State transition from RUNNING to PENDING_SHUTDOWN
2018-04-23 12:54:10 [INFO] [logger] StreamThread:336 - stream-thread [redacted-info-StreamThread-1] Informed to shut down
2018-04-23 12:54:10 [INFO] [logger] StreamThread:346 - stream-thread [redacted-info-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN
2018-04-23 12:54:10 [INFO] [logger] StreamThread:336 - stream-thread [redacted-info-StreamThread-2] Informed to shut down
2018-04-23 12:54:10 [INFO] [logger] StreamThread:346 - stream-thread [redacted-info-StreamThread-2] State transition from RUNNING to PENDING_SHUTDOWN
2018-04-23 12:54:10 [INFO] [logger] StreamThread:336 - stream-thread [redacted-info-StreamThread-3] Informed to shut down
2018-04-23 12:54:10 [INFO] [logger] StreamThread:346 - stream-thread [redacted-info-StreamThread-3] State transition from RUNNING to PENDING_SHUTDOWN
2018-04-23 12:54:10 [INFO] [logger] StreamThread:336 - stream-thread [redacted-info-StreamThread-4] Informed to shut down
2018-04-23 12:54:10 [INFO] [logger] StreamThread:346 - stream-thread [redacted-info-StreamThread-4] State transition from RUNNING to PENDING_SHUTDOWN
2018-04-23 12:54:10 [INFO] [logger] GlobalStreamThread:346 - global-stream-thread [redacted-info-GlobalStreamThread] State transition from RUNNING to PENDING_SHUTDOWN
2018-04-23 12:54:10 [INFO] [logger] StreamThread:336 - stream-thread [redacted-info-StreamThread-4] Shutting down
2018-04-23 12:54:10 [INFO] [logger] StreamThread:336 - stream-thread [redacted-info-StreamThread-1] Shutting down
2018-04-23 12:54:10 [INFO] [logger] GlobalStreamThread:336 - global-stream-thread [redacted-info-GlobalStreamThread] Shutting down
2018-04-23 12:54:10 [INFO] [logger] GlobalStreamThread:346 - global-stream-thread [redacted-info-GlobalStreamThread] State transition from PENDING_SHUTDOWN to DEAD
2018-04-23 12:54:10 [INFO] [logger] GlobalStreamThread:336 - global-stream-thread [redacted-info-GlobalStreamThread] Shutdown complete
2018-04-23 12:54:10 [INFO] [logger] StreamThread:336 - stream-thread [redacted-info-StreamThread-2] Shutting down

Issue Links

• links to: GitHub Pull Request #4950