[BEAM-2386] Change regex used for splitting words

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-py-core
• Labels: newbie starter
• Assignee: Unassigned
• Reporter: Ahmet Altay
• Created: 30/May/17 19:09
• Updated: 16/May/20 13:26
• Resolved: 14/Sep/19 21:56

Description

Regex used in splitting words ([A-Za-z\']+) only works on latin input, change it to make it work on non-latin inputs.

For example, see Java version:
https://github.com/apache/beam/blob/367fcb28d544934797d25cb34d54136b2d7d6e99/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java#L75