[SPARK-8418] Add single- and multi-value support to ML Transformers

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 17/Jun/15 23:57
• Updated: 21/May/19 04:12
• Resolved: 21/May/19 04:12

Description

It would be convenient if all feature transformers supported transforming columns of single values and multiple values, specifically:

• one column with one value (e.g., type Double)
• one column with multiple values (e.g., Array[Double] or Vector)

We could go as far as supporting multiple columns, but that may not be necessary since VectorAssembler could be used to handle that.

Estimators under ml.feature should also support this.

This will likely require a short design doc to describe:

• how input and output columns will be specified
• schema validation
• code sharing to reduce duplication

Issue Links

• contains: SPARK-20542 Add an API into Bucketizer that can bin a lot of columns all at once
• contains: SPARK-11215 Add multiple columns support to StringIndexer
• contains: SPARK-13030 Change OneHotEncoder to Estimator
• contains: SPARK-22397 Add multiple column support to QuantileDiscretizer