[LUCENE-4948] Stink bug in PostingsHighlighter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.3.1, 6.0
• Component/s: modules/highlighter
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Michael McCandless
• Created: 22/Apr/13 17:17
• Updated: 28/Aug/22 13:44
• Resolved: 22/Apr/13 18:12

Description

This test fail reproduces on IBM J9:

NOTE: reproduce with: ant test  -Dtestcase=TestPostingsHighlighter -Dtests.method=testCambridgeMA -Dtests.seed=2A9A93DAC39E0938 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=America/Yellowknife -Dtests.file.encoding=UTF-8

Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 37
        at __randomizedtesting.SeedInfo.seed([2A9A93DAC39E0938:AB8FF071AD305139]:0)
        at org.apache.lucene.search.postingshighlight.Passage.addMatch(Passage.java:53)
        at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightDoc(PostingsHighlighter.java:547)
        at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightField(PostingsHighlighter.java:425)
        at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:364)
        at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:268)
        at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlight(PostingsHighlighter.java:198)
        at org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testCambridgeMA(TestPostingsHighlighter.java:373)

I think it's because J9 grows arrays in a different progression than other JVMs ... we should fix PostingsHighlighter to forcefully grow the arrays to the same length instead of this:

    if (numMatches == matchStarts.length) {
      matchStarts = ArrayUtil.grow(matchStarts, numMatches+1);
      matchEnds = ArrayUtil.grow(matchEnds, numMatches+1);
      BytesRef newMatchTerms[] = new BytesRef[ArrayUtil.oversize(numMatches+1, RamUsageEstimator.NUM_BYTES_OBJECT_REF)];
      System.arraycopy(matchTerms, 0, newMatchTerms, 0, numMatches);
      matchTerms = newMatchTerms;
    }

Issue Links

• is duplicated by: LUCENE-4999 Lucene test (testCambridgeMA) fails when JVM 64-bit does not use memory compression