[SPARK-25114] RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.1.3, 2.2.2, 2.3.0, 2.3.1
• Fix Version/s: 2.2.3, 2.3.2, 2.4.0
• Component/s: Spark Core
• Labels: correctness
• Target Version/s: 2.3.2, 2.4.0
• Assignee: Xingbo Jiang
• Reporter: Xingbo Jiang
• Created: 14/Aug/18 14:26
• Updated: 02/Mar/20 08:01
• Resolved: 21/Aug/18 22:56

Description

It is possible for two objects to be unequal and yet we consider them as equal within RecordBinaryComparator, if the long values are separated by Int.MaxValue.

Issue Links

• is related to: SPARK-23207 Shuffle+Repartition on an DataFrame could lead to incorrect answers
• links to: [Github] Pull Request #22079 (bersprockets)
• links to: [Github] Pull Request #22101 (jiangxb1987)
• links to: [Github] Pull Request #22166 (jiangxb1987)
• links to: [Github] Pull Request #22211 (henryr)