[LUCENE-3911] improve BaseTokenStreamTestCase random string generation

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6, 4.0-ALPHA
• Fix Version/s: 3.6, 4.0-ALPHA
• Component/s: general/test
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 24/Mar/12 13:41
• Updated: 28/Aug/22 13:12
• Resolved: 24/Mar/12 15:43

Description

Most analysis tests use mocktokenizer (which splits on whitespace), but
its rare that we generate a string with 'many tokens'. So I think we should
try to generate more realistic test strings.