Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
Description
When we run a spark program with a projection for a array type, nullcheck at a call to write each element of an array is generated. If we know all of the elements do not have null at compilation time, we can eliminate code for nullcheck.
val df = sparkContext.parallelize(Seq(1.0, 2.0), 1).toDF("v") df.selectExpr("Array(v + 2.2, v + 3.3)").collect
Attachments
Issue Links
- duplicates
-
SPARK-16213 Reduce runtime overhead of a program that creates an primitive array in DataFrame
- Resolved
- is related to
-
SPARK-16070 DataFrame/Parquet issues with primitive arrays
- Resolved
- links to