[SPARK-35638] Introduce InternalField to manage dtypes and StructFields.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Takuya Ueshin
• Reporter: Takuya Ueshin
• Created: 03/Jun/21 22:11
• Updated: 12/Dec/22 18:10
• Resolved: 08/Jun/21 02:58

Description

Currently there are some performance issues in the pandas-on-Spark layer.

One of them is accessing Java DataFrame and run analysis phase too many times, especially just for retrieving the current column names or data types.

We should reduce the amount of unnecessary access.

Issue Links

• relates to: SPARK-34849 SPIP: Support pandas API layer on PySpark
• links to: [Github] Pull Request #32775 (ueshin)