Lucene - Core
  1. Lucene - Core
  2. LUCENE-6948

ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.10.4
    • Fix Version/s: 5.5, 6.0, 5.4.1
    • Component/s: core/search
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      With a very large index (in our case > 10G), we are seeing exceptions like:

      java.lang.ArrayIndexOutOfBoundsException: -62400
      at org.apache.lucene.util.PagedBytes$Reader.fill(PagedBytes.java:116)
      at org.apache.lucene.search.FieldCacheImpl$BinaryDocValuesImpl$1.get(FieldCacheImpl.java:1342)
      at org.apache.lucene.search.join.TermsCollector$SV.collect(TermsCollector.java:106)
      at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)
      at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
      at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
      at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
      at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

      The code in question is trying to allocate an array with a negative size. We believe the source of the error is in org.apache.lucene.search.FieldCacheImpl$BinaryDocValuesImpl$1.get where the following code occurs:

      final int pointer = (int) docToOffset.get(docID);
      if (pointer == 0)

      { term.length = 0; } else { bytes.fill(term, pointer); }

      The cast to int will break if the (long) result of docToOffset.get is too large, and is unnecessary in the first place since bytes.fill takes a long as its second parameter.

      Proposed fix:

      final long pointer = docToOffset.get(docID);
      if (pointer == 0) { term.length = 0; }

      else

      { bytes.fill(term, pointer); }
      1. LUCENE-6948.patch
        2 kB
        Christine Poerschke

        Activity

        Hide
        Christine Poerschke added a comment -

        Attaching patch against trunk with the fix proposed by Michael, plus two unrelated initialCapacity tweaks which I noticed whilst looking at code in the file.

        Show
        Christine Poerschke added a comment - Attaching patch against trunk with the fix proposed by Michael, plus two unrelated initialCapacity tweaks which I noticed whilst looking at code in the file.
        Hide
        Michael McCandless added a comment -

        +1 to the patch.

        Show
        Michael McCandless added a comment - +1 to the patch.
        Hide
        ASF subversion and git services added a comment -

        Commit 1723787 from Christine Poerschke in branch 'dev/trunk'
        [ https://svn.apache.org/r1723787 ]

        LUCENE-6948: Fix ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill by removing an unnecessary long-to-int cast. Also, unrelated, 2 ArrayList<>(initialCapacity) tweaks in getChildResources methods.

        Show
        ASF subversion and git services added a comment - Commit 1723787 from Christine Poerschke in branch 'dev/trunk' [ https://svn.apache.org/r1723787 ] LUCENE-6948 : Fix ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill by removing an unnecessary long-to-int cast. Also, unrelated, 2 ArrayList<>(initialCapacity) tweaks in getChildResources methods.
        Hide
        ASF subversion and git services added a comment -

        Commit 1723810 from Christine Poerschke in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1723810 ]

        LUCENE-6948: Fix ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill by removing an unnecessary long-to-int cast. Also, unrelated, 2 ArrayList<>(initialCapacity) tweaks in getChildResources methods. (merge in revision 1723787 from trunk)

        Show
        ASF subversion and git services added a comment - Commit 1723810 from Christine Poerschke in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1723810 ] LUCENE-6948 : Fix ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill by removing an unnecessary long-to-int cast. Also, unrelated, 2 ArrayList<>(initialCapacity) tweaks in getChildResources methods. (merge in revision 1723787 from trunk)
        Hide
        Christine Poerschke added a comment -

        Michael Lawley - thanks for the JIRA ticket and proposed fix. Michael McCandless - thanks for the patch review.

        Show
        Christine Poerschke added a comment - Michael Lawley - thanks for the JIRA ticket and proposed fix. Michael McCandless - thanks for the patch review.
        Hide
        Adrien Grand added a comment -

        Reopen for backport.

        Show
        Adrien Grand added a comment - Reopen for backport.
        Hide
        ASF subversion and git services added a comment -

        Commit 1724062 from Adrien Grand in branch 'dev/trunk'
        [ https://svn.apache.org/r1724062 ]

        LUCENE-6948: Move CHANGES entry to 5.4.1.

        Show
        ASF subversion and git services added a comment - Commit 1724062 from Adrien Grand in branch 'dev/trunk' [ https://svn.apache.org/r1724062 ] LUCENE-6948 : Move CHANGES entry to 5.4.1.
        Hide
        ASF subversion and git services added a comment -

        Commit 1724063 from Adrien Grand in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1724063 ]

        LUCENE-6948: Move CHANGES entry to 5.4.1.

        Show
        ASF subversion and git services added a comment - Commit 1724063 from Adrien Grand in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1724063 ] LUCENE-6948 : Move CHANGES entry to 5.4.1.
        Hide
        ASF subversion and git services added a comment -

        Commit 1724064 from Adrien Grand in branch 'dev/branches/lucene_solr_5_4'
        [ https://svn.apache.org/r1724064 ]

        LUCENE-6948: Fix ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill by removing an unnecessary long-to-int cast. Also, unrelated, 2 ArrayList<>(initialCapacity) tweaks in getChildResources methods.

        Show
        ASF subversion and git services added a comment - Commit 1724064 from Adrien Grand in branch 'dev/branches/lucene_solr_5_4' [ https://svn.apache.org/r1724064 ] LUCENE-6948 : Fix ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill by removing an unnecessary long-to-int cast. Also, unrelated, 2 ArrayList<>(initialCapacity) tweaks in getChildResources methods.

          People

          • Assignee:
            Christine Poerschke
            Reporter:
            Michael Lawley
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development