[SPARK-5888] Add OneHotEncoder as a Transformer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: ML
• Target Version/s: 1.4.0
• Assignee: Sandy Ryza
• Reporter: Xiangrui Meng
• Created: 18/Feb/15 19:12
• Updated: 28/May/15 04:34
• Resolved: 05/May/15 19:34

Description

`OneHotEncoder` takes a categorical column and output a vector column, which stores the category info in binaries.

val ohe = new OneHotEncoder()
  .setInputCol("countryIndex")
  .setOutputCol("countries")

It should read the category info from the metadata and assign feature names properly in the output column. We need to discuss the default naming scheme and whether we should let it process multiple categorical columns at the same time.

One category (the most frequent one) should be removed from the output to make the output columns linear independent. Or this could be an option tuned on by default.

Issue Links

• is duplicated by: SPARK-1216 Add a OneHotEncoder for handling categorical features
• is related to: SPARK-7912 Make OneHotEncoder more optimistic about the input
• relates to: SPARK-1216 Add a OneHotEncoder for handling categorical features
• relates to: SPARK-7579 User guide update for OneHotEncoder
• links to: [Github] Pull Request #5500 (sryza)