[KAFKA-9398] Kafka Streams main thread may not exit even after close timeout has passed

• Type: Improvement
• Status: Patch Available
• Priority: Critical
• Resolution: Unresolved
• Component/s: streams
• Assignee: Bill Bejeck
• Reporter: Bill Bejeck
• Created: 10/Jan/20 16:16
• Updated: 24/Feb/20 15:26

Description

Kafka Streams offers the KafkaStreams.close() method when shutting down a Kafka Streams application. There are two overloads to this method, one that takes no parameters and another taking a Duration specifying how long the close() method should block waiting for streams shut down operations to complete. The no-arg version of close() sets the timeout to Long.MAX_VALUE.

The issue is that if a StreamThread is taking to long to complete or if one of the Consumer or Producer clients is in a hung state, the Kafka Streams application won't exit even after the specified timeout has expired.

For example, consider this scenario:

1. A sink topic gets deleted by accident 
2. The user sets Producer max.block.ms config to a high value

In this case, the Producer will issue a WARN logging statement and will continue to make metadata requests looking for the expected topic. The Producer will continue making metadata requests up until the max.block.ms expires. If this value is high enough, calling close() with a timeout won't fix the issue as when the timeout expires, the Kafka Streams application's main thread won't exit.

To prevent this type of issue, we should call Thread.interrupt() on all StreamThread instances once the close() timeout has expired. 

Issue Links

• is related to: KAFKA-9416 Streams get stuck in `PENDING_SHUTDOWN` is underlying topics deleted.
• links to: GitHub Pull Request #7814