Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
3.0.0
-
None
-
None
Description
Spark 2.x has two separate implementations of the "create named struct" expression: regular CreateNamedStruct and CreateNamedStructUnsafe. The "unsafe" version was added in SPARK-9373 to support structs in GenerateUnsafeProjection. These two expressions both extend the CreateNameStructLike trait.
For Spark 3.0, I propose to always use the "unsafe" code path: this will avoid object allocation / boxing inefficiencies in the "safe" codepath, which is an especially big problem when generating Encoders for deeply-nested case classes.
Attachments
Issue Links
- links to