[SPARK-41757] Compatibility of string representation in Column

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Assignee: Hyukjin Kwon
• Reporter: Sandeep Singh
• Created: 29/Dec/22 01:44
• Updated: 26/Jan/23 22:05
• Resolved: 17/Jan/23 05:59

Description

Doctest in pyspark.sql.connect.column.Columnfails with the error below:

File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 120, in pyspark.sql.connect.column.Column
Failed example:
    df.name
Expected:
    Column<'name'>
Got:
    Column<'ColumnReference(name)'>
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 122, in pyspark.sql.connect.column.Column
Failed example:
    df["name"]
Expected:
    Column<'name'>
Got:
    Column<'ColumnReference(name)'>
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 127, in pyspark.sql.connect.column.Column
Failed example:
    df.age + 1
Expected:
    Column<'(age + 1)'>
Got:
    Column<'+(ColumnReference(age), Literal(1))'>
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 129, in pyspark.sql.connect.column.Column
Failed example:
    1 / df.age
Expected:
    Column<'(1 / age)'>
Got:
    Column<'/(Literal(1), ColumnReference(age))'> 

 

We should enable this back after fixing the issue in Spark Connect

Issue Links

• links to: [Github] Pull Request #39296 (grundprinzip)
• links to: [Github] Pull Request #39296 (grundprinzip)
• links to: [Github] Pull Request #39616 (HyukjinKwon)
• links to: [Github] Pull Request #39761 (techaddict)