Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11069

Add RegexTokenizer option to convert to lowercase

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: ML
    • Labels:
      None
    • Target Version/s:

      Description

      Tokenizer converts strings to lowercase automatically, but RegexTokenizer does not. It would be nice to add an option to RegexTokenizer to convert to lowercase. Proposal:

      • call the Boolean Param "toLowercase"
      • set default to false (so behavior does not change)

      Q: Should conversion to lowercase happen before or after regex matching?

      • Before: This is simpler.
      • After: This gives the user full control since they can have the regex treat upper/lower case differently.
        --> I'd vote for conversion before matching. If a user needs full control, they can convert to lowercase manually.

        Attachments

          Activity

            People

            • Assignee:
              yuhaoyan yuhao yang
              Reporter:
              josephkb Joseph K. Bradley
              Shepherd:
              Joseph K. Bradley
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: