[SPARK-19473] Several DataFrame Methods still fail with dot in column names

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Wayne Zhang
• Created: 06/Feb/17 07:44
• Updated: 01/Apr/17 10:45
• Resolved: 10/Feb/17 08:21

Description

Here is an example:

val df = Seq((1.0, 2.0), (2.0, 3.0)).toDF("y.a", "x.b")
df.select("y.a")
org.apache.spark.sql.AnalysisException: cannot resolve '`y.a`' given input columns: [y.a, x.b];;

df.withColumn("d", col("y.a") + col("x.b"))
org.apache.spark.sql.AnalysisException: cannot resolve '`y.a`' given input columns: [y.a, x.b];;

We can use backquote to avoid the errors, but this behavior is affecting some downstream work such as RFormula and SparkR.

Issue Links

• is duplicated by: SPARK-20182 Dot in DataFrame Column title causes errors