[SPARK-48555] Support Column type for several SQL functions in scala and python

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Connect, PySpark, Spark Core
• Labels: pull-request-available
• Assignee: Ron Serruya
• Reporter: Ron Serruya
• Created: 06/Jun/24 14:14
• Updated: 17/Jun/24 03:44
• Resolved: 17/Jun/24 00:31

Description

Currently, several SQL functions accept both native types and Columns, but only accept native types in their scala/python APIs:

• array_remove (works in SQL, scala, not in python)
• array_position(works in SQL, scala, not in python)
• map_contains_key (works in SQL, scala, not in python)
• substring (works only in SQL)

For example, this is possible in SQL:

spark.sql("select array_remove(col1, col2) from values(array(1,2,3), 2)")

But not in python:

df.select(F.array_remove(F.col("col1"), F.col("col2"))

Issue Links

• links to: GitHub Pull Request #46901