[SPARK-16042] Eliminate nullcheck code at projection for an array type

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Kazuaki Ishizaki
• Created: 18/Jun/16 06:17
• Updated: 12/Dec/22 18:10
• Resolved: 03/Feb/17 13:56

Description

When we run a spark program with a projection for a array type, nullcheck at a call to write each element of an array is generated. If we know all of the elements do not have null at compilation time, we can eliminate code for nullcheck.

val df = sparkContext.parallelize(Seq(1.0, 2.0), 1).toDF("v")
df.selectExpr("Array(v + 2.2, v + 3.3)").collect

Issue Links

• duplicates: SPARK-16213 Reduce runtime overhead of a program that creates an primitive array in DataFrame
• is related to: SPARK-16070 DataFrame/Parquet issues with primitive arrays
• links to: [Github] Pull Request #13757 (kiszk)