[SPARK-22979] Avoid per-record type dispatch in Python data conversion (EvaluatePython.fromJava)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: PySpark, SQL
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 06/Jan/18 08:35
• Updated: 12/Dec/22 17:50
• Resolved: 08/Jan/18 06:00

Description

Seems we are type dispatching between Java objects (from Pyrolite) to Spark's internal data format.

See https://github.com/apache/spark/blob/3f958a99921d149fb9fdf7ba7e78957afdad1405/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala#L89-L162

Looks we can make converters each for each type and then reuse it.

Issue Links

• links to: [Github] Pull Request #20172 (HyukjinKwon)