[LUCENE-9286] FST arc.copyOf clones BitTables and this can lead to excessive memory use

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.5
• Fix Version/s: 8.6
• Lucene Fields: New
• Assignee: Bruno Roustant
• Reporter: Dawid Weiss
• Created: 21/Mar/20 10:30
• Updated: 15/Jul/20 15:09
• Resolved: 09/Apr/20 10:20

Description

I see a dramatic increase in the amount of memory required for construction of (arguably large) automata. It currently OOMs with 8GB of memory consumed for bit tables. I am pretty sure this didn't require so much memory before (the automaton is ~50MB after construction).

Something bad happened in between. Thoughts, Bruno Roustant, Michael Sokolov?

Issue Links

• is related to: LUCENE-8920 Reduce size of FSTs due to use of direct-addressing encoding
• links to: GitHub Pull Request #1416
• links to: GitHub Pull Request #1429