[SPARK-23989] When using `SortShuffleWriter`, the data will be overwritten

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: Spark Core
• Assignee: Wenchen Fan
• Reporter: liuxian
• Created: 16/Apr/18 08:48
• Updated: 19/Apr/18 15:57
• Resolved: 19/Apr/18 15:57

Description

When using `SortShuffleWriter`, we only insert  'AnyRef' into 'PartitionedAppendOnlyMap' or 'PartitionedPairBuffer'.

For this function:

override def write(records: Iterator[Product2[K, V]])

the value of 'records' is `UnsafeRow`, so  the value will be overwritten

 

Issue Links

• links to: [Github] Pull Request #21101 (cloud-fan)