[SPARK-20416] Column names inconsistent for UDFs in SQL vs Dataset

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Takeshi Yamamuro
• Reporter: Jacek Laskowski
• Created: 20/Apr/17 19:08
• Updated: 11/May/17 16:50
• Resolved: 11/May/17 16:50

Description

As you can see below, the name of the columns in SQL vs Dataset is different.

scala> val timesTwoUDF = spark.udf.register("timesTwo", (x: Int) => x * 2)
timesTwoUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,Some(List(IntegerType)))

scala> spark.sql("SELECT timesTwo(1)").show
+---------------+
|UDF:timesTwo(1)|
+---------------+
|              2|
+---------------+

scala> spark.range(1, 2).toDF("x").select(timesTwoUDF($"x")).show
+------+
|UDF(x)|
+------+
|     2|
+------+

Issue Links

• links to: [Github] Pull Request #17712 (maropu)