[SPARK-23465] Dataset.withAllColumnsRenamed should map all column names to a new one

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.3.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Mihaly Toth
• Created: 19/Feb/18 16:16
• Updated: 05/Apr/18 15:07
• Resolved: 05/Apr/18 15:07

Description

Currently one can only rename a column only one by one using withColumnRenamed() function. When one would like to rename all or most of the columns it would be easier to specify an algorithm for mapping from the old to the new name (like prefixing) than iterating over all the fields.

Example usage is joining to a Dataset with the same or similar schema (special case is self joining) where the names are the same or overlapping. Such a joined Dataset would fail at saveAsTable()

With the new function usage would be easy like that:

ds.withAllColumnsRenamed("prefix" + _)

Issue Links

• links to: [Github] Pull Request #20674 (misutoth)