[LUCENE-6046] RegExp.toAutomaton high memory use

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.10.1
• Fix Version/s: 4.10.3, 5.0, 6.0
• Component/s: core/queryparser
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Lee Hinman
• Created: 03/Nov/14 09:28
• Updated: 28/Aug/22 14:19
• Resolved: 04/Nov/14 22:29

Description

When creating an automaton from an org.apache.lucene.util.automaton.RegExp, it's possible for the automaton to use so much memory it exceeds the maximum array size for java.

The following caused an OutOfMemoryError with a 32gb heap:

new RegExp("\\[\\[(Datei|File|Bild|Image):[^]]*alt=[^]|}]{50,200}").toAutomaton();

When increased to a 60gb heap, the following exception is thrown:

  1> java.lang.IllegalArgumentException: requested array size 2147483624 exceeds maximum array in java (2147483623)
  1>     __randomizedtesting.SeedInfo.seed([7BE81EF678615C32:95C8057A4ABA5B52]:0)
  1>     org.apache.lucene.util.ArrayUtil.oversize(ArrayUtil.java:168)
  1>     org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:295)
  1>     org.apache.lucene.util.automaton.Automaton$Builder.addTransition(Automaton.java:639)
  1>     org.apache.lucene.util.automaton.Operations.determinize(Operations.java:741)
  1>     org.apache.lucene.util.automaton.MinimizationOperations.minimizeHopcroft(MinimizationOperations.java:62)
  1>     org.apache.lucene.util.automaton.MinimizationOperations.minimize(MinimizationOperations.java:51)
  1>     org.apache.lucene.util.automaton.RegExp.toAutomaton(RegExp.java:477)
  1>     org.apache.lucene.util.automaton.RegExp.toAutomaton(RegExp.java:426)