[SPARK-14659] OneHotEncoder support drop first category alphabetically in the encoded vector

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: ML
• Target Version/s: 2.3.0
• Assignee: Wayne Zhang
• Reporter: Yanbo Liang
• Shepherd: Yanbo Liang
• Created: 15/Apr/16 12:53
• Updated: 26/May/17 05:45
• Resolved: 26/May/17 05:45

Description

R formula drop the first category alphabetically when encode string/category feature. Spark RFormula use OneHotEncoder to encode string/category feature into vector, but only supporting "dropLast" by string/category frequencies. This will cause SparkR produce different models compared with native R.

Issue Links

• Is contained by: SPARK-15540 RFormula and R feature processing improvement umbrella
• relates to: SPARK-14657 RFormula output wrong features when formula w/o intercept
• links to: [Github] Pull Request #17005 (actuaryzhang)
• links to: [Github] Pull Request #17967 (actuaryzhang)