[SPARK-29033] Always use CreateNamedStructUnsafe, the UnsafeRow-based version of the CreateNamedStruct codepath

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 10/Sep/19 07:40
• Updated: 29/Oct/19 07:53
• Resolved: 26/Oct/19 04:46

Description

Spark 2.x has two separate implementations of the "create named struct" expression: regular CreateNamedStruct and CreateNamedStructUnsafe. The "unsafe" version was added in SPARK-9373 to support structs in GenerateUnsafeProjection. These two expressions both extend the CreateNameStructLike trait.

For Spark 3.0, I propose to always use the "unsafe" code path: this will avoid object allocation / boxing inefficiencies in the "safe" codepath, which is an especially big problem when generating Encoders for deeply-nested case classes.

Issue Links

• links to: GitHub Pull Request #25745