[SPARK-8169] Add StopWordsRemover as a transformer

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: ML
• Target Version/s: 1.5.0
• Assignee: yuhao yang
• Reporter: Xiangrui Meng
• Shepherd: Xiangrui Meng
• Created: 08/Jun/15 20:52
• Updated: 12/Aug/15 20:29
• Resolved: 01/Aug/15 09:31

Description

StopWordsRemover takes a string array column and outputs a string array column with all defined stop words removed. The transformer should also come with a standard set of stop words as default.

val stopWords = new StopWordsRemover()
  .setInputCol("words")
  .setOutputCol("cleanWords")
  .setStopWords(Array(...)) // optional
val output = stopWords.transform(df)

Issue Links

• is related to: SPARK-9679 Add python interface for ml.feature.StopWordsRemover
• is related to: SPARK-9680 Update programming guide section for ml.feature.StopWordsRemover
• is required by: SPARK-5571 LDA should handle text as well
• relates to: SPARK-9892 User Guide for StopWordsRemover
• links to: [Github] Pull Request #6742 (hhbyyh)