[LUCENE-9186] remove linefiledocs usage from basetokenstreamtestcase

• Type: Task
• Status: Patch Available
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.5
• Component/s: general/test
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 28/Jan/20 11:17
• Updated: 22/Nov/24 14:44
• Resolved: 28/Jan/20 16:56

Description

LineFileDocs is slow, even to open. That's because it (very slowly) "skips" to a pseudorandom position into a 5MB gzip stream when you open it.

There was a time when we didn't have a nice string generator for tests (TestUtil.randomAnalysisString), but now we do. And when it was introduced it found interesting new things that linefiledocs never found.

This speeds up all the analyzer tests.