[SPARK-20537] OffHeapColumnVector reallocation may not copy existing data

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Kazuaki Ishizaki
• Reporter: Kazuaki Ishizaki
• Created: 30/Apr/17 10:16
• Updated: 11/May/17 20:02
• Resolved: 02/May/17 05:58

Description

As SPARK-20474 revealed, reallocation in OnHeapColumnVector may copy a part of the original storage.
OffHeapColumnVector reallocation also copies to the new storage data up to elementsAppended. This variable is only updated when using the ColumnVector.appendX API, while ColumnVector.putX is more commonly used.

Issue Links

• links to: [Github] Pull Request #17811 (kiszk)