[SPARK-40414] Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: PySpark
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 13/Sep/22 12:53
• Updated: 14/Sep/22 05:18
• Resolved: 14/Sep/22 05:18

Description

During the work of flatMapGroupsWithState in PySpark, we figured out that we are unable to reuse PythonArrowInput and PythonArrowOutput, as PythonArrowInput and PythonArrowOutput are too specific to the strict input data (row) and output data.

To reuse the implementations we should make these traits more general to handle more generic type of data.

Issue Links

• links to: [Github] Pull Request #37864 (HeartSaVioR)