[KAFKA-9416] Streams get stuck in `PENDING_SHUTDOWN` is underlying topics deleted.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Component/s: streams
• Assignee: Unassigned
• Reporter: Andy Coates
• Created: 13/Jan/20 18:27
• Updated: 13/Jan/20 18:30

Description

We've noticed that if topics are deleted from under a running topology, e.g. repartition, changelog or sink topics, then the stream threads transition from `RUNNING` to `PENDING_SHUTDOWN`, but then do not transition to `ERROR`.

Likewise, if a Kafka cluster has auto topic create disabled and a topology is started where its sink topic(s) do not exist, then the topology similarly gets stuck in `PENDING_SHUTDOWN`.

Once the query is stuck in `PENDING_SHUTDOWN` any call to close the topology blocks, as per https://issues.apache.org/jira/browse/KAFKA-9398.

We would like to see Kafka Streams handle this case and correctly transition to `ERROR` state.

The root cause of this looks to be that we have a large `max.block.ms` set on the KS Producer and consumer.  The producer can get stuck trying to produce to a topic that does not exist. Likewise, a consumer can get stuck trying to commit offsets for a topic that does not exist.  

The cause of this is covered in more detail here: https://github.com/confluentinc/ksql/issues/4268#issuecomment-573020281

Issue Links

• relates to: KAFKA-9398 Kafka Streams main thread may not exit even after close timeout has passed