Lucene - Core
  1. Lucene - Core
  2. LUCENE-4547

DocValues field broken on large indexes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2, 5.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I tried to write a test to sanity check LUCENE-4536 (first running against svn revision 1406416, before the change).

      But i found docvalues is already broken here for large indexes that have a PackedLongDocValues field:

      final int numDocs = 500000000;
      for (int i = 0; i < numDocs; ++i) {
        if (i == 0) {
          field.setLongValue(0L); // force > 32bit deltas
        } else {
          field.setLongValue(1<<33L); 
        }
        w.addDocument(doc);
      }
      w.forceMerge(1);
      w.close();
      dir.close(); // checkindex
      
      [junit4:junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #0,6,TGRP-Test2GBDocValues]
      [junit4:junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException: -65536
      [junit4:junit4]   2> 	at __randomizedtesting.SeedInfo.seed([5DC54DB14FA5979]:0)
      [junit4:junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:535)
      [junit4:junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:508)
      [junit4:junit4]   2> Caused by: java.lang.ArrayIndexOutOfBoundsException: -65536
      [junit4:junit4]   2> 	at org.apache.lucene.util.ByteBlockPool.deref(ByteBlockPool.java:305)
      [junit4:junit4]   2> 	at org.apache.lucene.codecs.lucene40.values.FixedStraightBytesImpl$FixedBytesWriterBase.set(FixedStraightBytesImpl.java:115)
      [junit4:junit4]   2> 	at org.apache.lucene.codecs.lucene40.values.PackedIntValues$PackedIntsWriter.writePackedInts(PackedIntValues.java:109)
      [junit4:junit4]   2> 	at org.apache.lucene.codecs.lucene40.values.PackedIntValues$PackedIntsWriter.finish(PackedIntValues.java:80)
      [junit4:junit4]   2> 	at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:130)
      [junit4:junit4]   2> 	at org.apache.lucene.codecs.PerDocConsumer.merge(PerDocConsumer.java:65)
      
      1. test.patch
        3 kB
        Robert Muir
      2. LUCENE-4547.patch
        1.86 MB
        Robert Muir

        Issue Links

          Activity

          Yonik Seeley made changes -
          Link This issue breaks SOLR-5231 [ SOLR-5231 ]
          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Robert Muir made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Robert Muir made changes -
          Link This issue incorporates LUCENE-3729 [ LUCENE-3729 ]
          Robert Muir made changes -
          Link This issue incorporates LUCENE-4033 [ LUCENE-4033 ]
          Robert Muir made changes -
          Link This issue incorporates LUCENE-4035 [ LUCENE-4035 ]
          Robert Muir made changes -
          Attachment LUCENE-4547.patch [ 12568149 ]
          Robert Muir made changes -
          Link This issue incorporates LUCENE-4717 [ LUCENE-4717 ]
          Robert Muir made changes -
          Link This issue incorporates LUCENE-4540 [ LUCENE-4540 ]
          Robert Muir made changes -
          Priority Blocker [ 1 ] Major [ 3 ]
          Mark Miller made changes -
          Fix Version/s 4.2 [ 12323899 ]
          Fix Version/s 4.1 [ 12321140 ]
          Mark Miller made changes -
          Fix Version/s 5.0 [ 12321663 ]
          Adrien Grand made changes -
          Link This issue blocks SOLR-3855 [ SOLR-3855 ]
          Robert Muir made changes -
          Summary PackedIntsDocValue field broken on large indexes DocValues field broken on large indexes
          Robert Muir made changes -
          Field Original Value New Value
          Attachment test.patch [ 12552474 ]
          Robert Muir created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Robert Muir
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development