[SPARK-45130] Avoid Spark connect ML model to change input pandas dataframe

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, ML, PySpark
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Weichen Xu
• Reporter: Weichen Xu
• Created: 12/Sep/23 08:31
• Updated: 18/Sep/23 04:54
• Resolved: 18/Sep/23 04:54

Description

Currently, to avoid data copy, Spark connect ML model directly changes input pandas dataframe for appending prediction columns. But we can use `pandas_df.copy(deep=False)` to shallow copy it and then append prediction columns in copied dataframe. This is easier for user to use it.

Issue Links

• links to: GitHub Pull Request #42887