[SPARK-11069] Add RegexTokenizer option to convert to lowercase

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: ML
• Target Version/s: 1.6.0
• Assignee: yuhao yang
• Reporter: Joseph K. Bradley
• Shepherd: Joseph K. Bradley
• Created: 12/Oct/15 19:19
• Updated: 05/Jul/17 14:39
• Resolved: 10/Nov/15 00:57

Description

Tokenizer converts strings to lowercase automatically, but RegexTokenizer does not. It would be nice to add an option to RegexTokenizer to convert to lowercase. Proposal:

• call the Boolean Param "toLowercase"
• set default to false (so behavior does not change)

Q: Should conversion to lowercase happen before or after regex matching?

• Before: This is simpler.
• After: This gives the user full control since they can have the regex treat upper/lower case differently.
  --> I'd vote for conversion before matching. If a user needs full control, they can convert to lowercase manually.

Issue Links

• links to: [Github] Pull Request #9092 (hhbyyh)