Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
New
Description
LowerCaseTokenizer combines tokenization and filtering in a way that prevents us improving the normalization API. We should deprecate and remove it, as it can be replaced simply with a LetterTokenizer and LowerCaseFilter.