[SPARK-17490] Optimize SerializeFromObject for primitive array

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: SQL
• External issue ID: SPARK-16043
• Assignee: Kazuaki Ishizaki
• Reporter: Kazuaki Ishizaki
• Created: 10/Sep/16 06:37
• Updated: 08/Nov/16 20:49
• Resolved: 07/Nov/16 23:17

Description

In logical plan, SerializeFromObject for an array always use GenericArrayData as a destination. UnsafeArrayData could be used for an primitive array. This is a simple approach to solve issues that are addressed by SPARK-16043.

Here is a motivating example.

sparkContext.parallelize(Seq(Array(1)), 1).toDS.map(e => e).show

Issue Links

• is duplicated by: SPARK-16215 Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset
• links to: [Github] Pull Request #15044 (kiszk)