[SPARK-31690] Backport pyspark Interaction to Spark 2.4.x

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.4.5
• Component/s: ML, PySpark
• Assignee: Unassigned
• Reporter: Luca Giovagnoli
• Created: 12/May/20 19:25
• Updated: 12/Dec/22 18:10
• Resolved: 13/May/20 08:05

Description

In our company, we could really make use of the Interaction pyspark wrapper on spark 2.4.x.

"Interaction" is available on spark 3.0, so I'm proposing to backport the following code to the current Spark2.4.6-rc1:
- https://issues.apache.org/jira/browse/SPARK-26970
- https://github.com/apache/spark/pull/24426/files

 

I'm available to pick this up if it's approved.