[STORM-2733] Make Load Aware Shuffle much better at really bad situations

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 2.0.0
• Fix Version/s: 2.0.0
• Component/s: storm-client
• Labels: pull-request-available
• Assignee: Robert Joseph Evans
• Reporter: Robert Joseph Evans
• Created: 08/Sep/17 16:32
• Updated: 14/Sep/17 15:33
• Resolved: 14/Sep/17 15:32

Description

We recently had an issue where some bolts got really backed up and started to die from OOMs. The issue ended up being 2 fold.

First the GC really slowed down the worker so much that it could not keep up even with < 1% of the traffic that was still being sent to it. Which made it almost impossible to recover.

The second issue was that the serialization of the tuples took a lot longer than the processing, which resulted in the send queue filling up much more quickly than the receive queue.

To help fix this issue I plan to address this in 2 ways. First we need a better algorithm that can actually shut off the flow entirely to a very slow bolt and second we need to take the send queue into account when shuffling.

This is not a full set of changes needed by STORM-2686 but it is a step in that direction. I am going to try and set it up so that the two algorithms would work nicely together.

Issue Links

• links to: GitHub Pull Request #2321