[SPARK-21536] Remove the workaroud to allow dots in field names in R's createDataFame

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: SparkR
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Hyukjin Kwon
• Created: 26/Jul/17 06:20
• Updated: 12/Dec/22 18:11
• Resolved: 08/Oct/19 05:42

Description

We are currently converting dots to underscore in some cases as below in SparkR

> createDataFrame(list(list(1)), "a.a")
SparkDataFrame[a_a:double]
...
In FUN(X[[i]], ...) : Use a_a instead of a.a  as column name

> createDataFrame(list(list(a.a = 1)))
SparkDataFrame[a_a:double]
...
In FUN(X[[i]], ...) : Use a_a instead of a.a  as column name

This looks introduced in the first place due to SPARK-2775 but now it is fixed in SPARK-6898.

Issue Links

• relates to: SPARK-12191 Support "." character in DataFrame column name for ML
• links to: [Github] Pull Request #18737 (HyukjinKwon)