[SPARK-6634] Allow replacing columns in Transformers

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.3.0
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 31/Mar/15 17:19
• Updated: 21/May/19 04:17
• Resolved: 21/May/19 04:17

Description

Currently, Transformers do not allow input and output columns to share the same name. (In fact, this is not allowed but also not even checked.)

Short-term proposal: Disallow input and output columns with the same name, and add a check in transformSchema.

Long-term proposal: Allow input & output columns with the same name, and where the behavior is that the output columns replace input columns with the same name.

Issue Links

• is related to: SPARK-6635 DataFrame.withColumn can create columns with identical names