[SPARK-24853] Support Column type for withColumn and withColumnRenamed apis

• Type: Improvement
• Status: Reopened
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.2.2, 3.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: nirav patel
• Created: 18/Jul/18 18:53
• Updated: 12/Dec/22 18:10

Description

Can we add overloaded version of withColumn or withColumnRenamed that accept Column type instead of String? That way I can specify FQN in case when there is duplicate column names. e.g. if I have 2 columns with same name as a result of join and I want to rename one of the field I can do it with this new API.
 
This would be similar to Drop api which supports both String and Column type.
 
def
withColumn(colName: Column, col: Column): DataFrame
Returns a new Dataset by adding a column or replacing the existing column that has the same name.
 
def
withColumnRenamed(existingName: Column, newName: Column): DataFrame
Returns a new Dataset with a column renamed.
 
 
 
I think there should also be this one:
 
def
withColumnRenamed(existingName: Column, newName: Column): DataFrame
Returns a new Dataset with a column renamed.