[LUCENE-6814] PatternTokenizer indefinitely holds heap equal to max field it has ever tokenized

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.4, 6.0
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 24/Sep/15 08:50
• Updated: 28/Aug/22 14:43
• Resolved: 05/Nov/15 20:11

Description

Caught by Alex Chow in this Elasticsearch issue: https://github.com/elastic/elasticsearch/issues/13721

Today, PatternTokenizer reuses a single StringBuilder, but it doesn't free its heap usage after tokenizing is done. We can either stop reusing, or ask it to .trimToSize when we are done ...