[SPARK-23166] Add maxDF Parameter to CountVectorizer

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.1
• Fix Version/s: 2.4.0
• Component/s: ML
• Assignee: Yacine Mazari
• Reporter: Yacine Mazari
• Created: 21/Jan/18 02:50
• Updated: 06/Mar/18 23:15
• Resolved: 28/Jan/18 16:28

Description

Currently, the CountVectorizer has a minDF parameter.

It might be useful to also have a maxDF parameter.
It will be used as a threshold for filtering all the terms that occur very frequently in a text corpus, because they are not very informative or could even be stop-words.

This is analogous to scikit-learn, CountVectorizer, max_df.

Issue Links

• relates to: SPARK-23615 Add maxDF Parameter to Python CountVectorizer
• links to: [Github] Pull Request #20364 (mgaido91)
• links to: [Github] Pull Request #20366 (ymazari)
• links to: [Github] Pull Request #20367 (ymazari)