[SPARK-6914] DataFrame.withColumn should take metadata

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Component/s: SQL
• Target Version/s: 1.6.0
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 14/Apr/15 22:47
• Updated: 13/Aug/15 22:44
• Resolved: 13/Aug/15 22:44

Description

It would be nice if DataFrame.withColumn could take metadata. Currently (as far as I can tell), the best way to specify it is:

dataset.withColumn(columnName, newCol.as(columnName, myMetadata))

where "newCol" is a new column (in my use case, a column created by a UDF).

This is weird since the columnName is specified twice.

Issue Links

• is duplicated by: SPARK-9929 support adding metadata in withColumn