[STORM-3637] Looping topology structure can cause backpressure to deadlock

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0, 2.1.0
• Component/s: storm-core
• Assignee: Unassigned
• Reporter: Simon Cooper
• Created: 14/May/20 13:41
• Updated: 11/Aug/20 10:52

Description

When you have a topology structure with loops in it (BoltA and BoltB send tuples to each other), it can cause backpressure to deadlock.

The scenario is that BoltA suddenly takes a long time to process a tuple (in our situation, it's doing a database operation). This causes the task input queue to fill up, setting the backpressure flag.

BoltB, which is sending a tuple to BoltA, then cannot send, and the tuple is held in the emit queue. This blocks any tuples behind it, and also stops BoltB from executing. This means the input queue to BoltB will build up, until that backpressure flag is also set - and then when BoltA next wants to send a tuple to BoltB, it will irrevocably deadlock.