[SPARK-36554] Error message while trying to use spark sql functions directly on dataframe columns without using select expression

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.3.0
• Component/s: Documentation, Examples, PySpark
• Labels: documentation features functions spark-sql
• Epic Link: New functions in Spark SQL 3.3

Language

Python

• Assignee: Nicolas Azrak
• Reporter: Lekshmi Ramachandran
• Shepherd: Lekshmi Ramachandran
• Created: 22/Aug/21 02:29
• Updated: 12/Dec/22 18:10
• Resolved: 02/Nov/21 01:53

Description

The below code generates a dataframe successfully . Here make_date function is used inside a select expression

 

from pyspark.sql.functions import  expr, make_date

df = spark.createDataFrame([(2020, 6, 26), (1000, 2, 29), (-44, 1, 1)],['Y', 'M', 'D'])

df.select("*",expr("make_date(Y,M,D) as lk")).show()

 

The below code fails with a message "cannot import name 'make_date' from 'pyspark.sql.functions'" . Here the make_date function is directly called on dataframe columns without select expression

 

from pyspark.sql.functions import make_date

df = spark.createDataFrame([(2020, 6, 26), (1000, 2, 29), (-44, 1, 1)],['Y', 'M', 'D'])

df.select(make_date(df.Y,df.M,df.D).alias("datefield")).show()

 
The error message generated is misleading when it says "cannot import make_date from pyspark.sql.functions"

 

Issue Links

• links to: [Github] Pull Request #34356 (nicolasazrak)