[STORM-2210] ShuffleGrouping does not produce even distribution

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.2
• Fix Version/s: 2.0.0, 1.1.0, 1.0.3
• Component/s: storm-core
• Epic Link: Release Apache Storm 1.1.0
• Assignee: Kevin Peek
• Reporter: Kevin Peek
• Created: 21/Nov/16 17:45
• Updated: 23/Nov/16 19:41
• Resolved: 23/Nov/16 19:41

Description

When testing the ShuffleGrouping in a multithreaded environment, it produces an extremely uneven distribution.

This appears to be a result of the Collection.shuffle call here. https://github.com/apache/storm/blob/1.0.x-branch/storm-core/src/jvm/org/apache/storm/grouping/ShuffleGrouping.java#L58

Because current was set to zero before the shuffle, other threads are able to access the arrayList while it is being shuffled.

Stephen's gist here includes a test that results in a very uneven distribution of taskIds from the ShuffleGrouping: https://gist.github.com/Crim/61537958df65a5e13b3844b2d5e28cde

I would have expected the taskIds from the ShuffleGrouping to be almost uniformly distributed.

Issue Links

• links to: GitHub Pull Request #1790
• links to: GitHub Pull Request #1797