[SPARK-46082] Fix protobuf string representation for Pandas Functions API with Spark Connect

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, PySpark
• Labels: pull-request-available
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 24/Nov/23 01:07
• Updated: 24/Nov/23 03:02
• Resolved: 24/Nov/23 03:02

Description

df = spark.range(1)
df.mapInPandas(lambda x: x, df.schema)._plan.print()

prints as below. It should includes functions.

<MapPartitions is_barrier='False'>
  <Range start='0', end='1', step='1', num_partitions='None'>

Issue Links

• links to: GitHub Pull Request #43991