[MAHOUT-688] High Document Frequency pruning for seq2sparse

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.6
• Labels: Vectorization
• Assignee: Grant Ingersoll
• Reporter: Vasil Vasilev
• Created: 05/May/11 13:02
• Updated: 09/Feb/12 14:00
• Resolved: 08/Dec/11 11:12

Description

This improvement allows to prune the words with high document frequencies from the tf and tf-idf vectors produced by seq2sparse, based on the standard deviation of the words' document frequencies and specifying which rods to be pruned in a means of times this standard deviation. One good option is 3 times the standard deviation