[HIVE-13108] Operators: SORT BY randomness is not safe with network partitions

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.1, 1.3.0, 2.0.0, 2.0.1
• Fix Version/s: 1.3.0, 2.0.1, 2.1.0
• Component/s: Spark, Tez
• Target Version/s: 2.0.1
• Release Note: Operators: SORT BY randomness is not safe with network partitions (Gopal V, reviewed by Sergey Shelukhin)
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 20/Feb/16 02:52
• Updated: 29/May/18 17:08
• Resolved: 03/Mar/16 06:55

Description

SORT BY relies on a transient Random object, which is initialized once per deserialize operation.

This results in complications during a network partition and when Tez/Spark reuses a cached plan.

Issue Links

• relates to: HIVE-19671 Distribute by rand() can lead to data inconsistency