[KAFKA-9562] Streams not making progress under heavy failures with EOS enabled on 2.5 branch

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Component/s: streams
• Assignee: Boyang Chen
• Reporter: John Roesler
• Created: 15/Feb/20 04:13
• Updated: 10/Mar/20 17:52
• Resolved: 10/Mar/20 17:52

Description

During soak testing in preparation for the 2.5.0 release, we have discovered a case in which Streams appears to stop making progress. Specifically, this is a failure-resilience test in which we inject network faults separating the instances from the brokers roughly every twenty minutes.

On 2.4, Streams would obviously spend a lot of time rebalancing under this scenario, but would still make progress. However, on the current 2.5 branch, Streams effectively stops making progress except rarely.

This appears to be a severe regression, so I'm filing this ticket as a 2.5.0 release blocker.

Issue Links

• links to: GitHub Pull Request #8116

Sub-Tasks

• Deprecate rebalanceException on StreamThread to avoid infinite loop: Resolved
• Ensure EOS task producer not through fatal error in unclean shutdown: Resolved