[SPARK-10105] Adding most k frequent words parameter to Word2Vec implementation

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Component/s: MLlib
• Labels: bulk-closed mllib top-k word2vec
• Assignee: Unassigned
• Reporter: Antonio Murgia
• Created: 19/Aug/15 03:09
• Updated: 21/May/19 04:35
• Resolved: 21/May/19 04:35

Description

When training Word2Vec on a really big dataset, it's really hard to evaluate the right minCount parameter, it would really help having a parameter to choose how many words you want to be in the vocabulary.
Furthermore, the original Word2Vec paper, state that they took into account the most frequent 1M words.

Issue Links

• links to: [Github] Pull Request #8301 (tmnd1991)