[SPARK-18374] Incorrect words in StopWords/english.txt

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.1
• Fix Version/s: 2.2.0
• Component/s: ML
• Labels: releasenotes
• Assignee: yuhao yang
• Reporter: nirav patel
• Created: 09/Nov/16 02:08
• Updated: 15/Dec/16 01:31
• Resolved: 06/Dec/16 21:12

Description

I was just double checking english.txt for list of stopwords as I felt it was taking out valid tokens like 'won'. I think issue is english.txt list is missing apostrophe character and all character after apostrophe. So "won't" becam "won" in that list; "wouldn't" is "wouldn" .

Here are some incorrect tokens in this list:

won
wouldn
ma
mightn
mustn
needn
shan
shouldn
wasn
weren

I think ideal list should have both style. i.e. won't and wont both should be part of english.txt as some tokenizer might remove special characters. But 'won' is obviously shouldn't be in this list.

Here's list of snowball english stop words:
http://snowball.tartarus.org/algorithms/english/stop.txt

Issue Links

• links to: [Github] Pull Request #16103 (hhbyyh)