[SPARK-45852] Gracefully deal with recursion exception during Spark Connect logging

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: pull-request-available
• Assignee: Martin Grund
• Reporter: Martin Grund
• Created: 09/Nov/23 11:44
• Updated: 20/Nov/23 06:14
• Resolved: 10/Nov/23 08:12

Description

```
from google.protobuf.text_format import MessageToString
from pyspark.sql.functions import col, lit

df = spark.range(10)

for x in range(800):
df = df.withColumn(f"next

{x}

", lit(1))
MessageToString(df._plan.to_proto(spark._client), as_one_line=True)

```

Issue Links

• is related to: SPARK-45987 Fix `pyspark.sql.tests.connect.test_connect_basic` in Python 3.11
• is related to: SPARK-45990 Upgrade `protobuf` to 4.25.1 to support `Python 3.11`
• links to: GitHub Pull Request #43732