[SPARK-12988] Can't drop columns that contain dots

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.0.0
• Component/s: SQL
• Target Version/s: 1.6.2, 2.0.0
• Assignee: Xiang Zhong
• Reporter: Michael Armbrust
• Created: 25/Jan/16 21:06
• Updated: 01/Jun/16 02:52
• Resolved: 01/Jun/16 02:52

Description

Neither of theses works:

val df = Seq((1, 1)).toDF("a_b", "a.c")
df.drop("a.c").collect()

df: org.apache.spark.sql.DataFrame = [a_b: int, a.c: int]

val df = Seq((1, 1)).toDF("a_b", "a.c")
df.drop("`a.c`").collect()

df: org.apache.spark.sql.DataFrame = [a_b: int, a.c: int]

Given that you can't use drop to drop subfields, it seems to me that we should treat the column name literally (i.e. as though it is wrapped in back ticks).

Issue Links

• is duplicated by: SPARK-12987 Drop fails when columns contain dots
• is duplicated by: SPARK-13455 Periods in dataframe column names breaks df.drop(<string>)
• relates to: SPARK-13197 When trying to select from the data frame which contains the columns with . in it, it is throwing exception.
• links to: [Github] Pull Request #10943 (dilipbiswal)
• links to: [Github] Pull Request #13306 (clockfly)