Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6062

Index corruption from numeric DV updates

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 5.0, 6.0
    • None
    • None
    • New

    Description

      I hit this while working on on LUCENE-6005: when cutting over TestNumericDocValuesUpdates to the new Document2 API, I accidentally enabled additional docValues in the test, and this this:

      There was 1 failure:
      1) testUpdateSegmentWithNoDocValues(org.apache.lucene.index.TestNumericDocValuesUpdates)
      java.io.FileNotFoundException: _1_Asserting_0.dvm in dir=RAMDirectory@259847e5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30981eab
      	at __randomizedtesting.SeedInfo.seed([0:7C88A439A551C47D]:0)
      	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:645)
      	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
      	at org.apache.lucene.codecs.lucene50.Lucene50DocValuesProducer.<init>(Lucene50DocValuesProducer.java:130)
      	at org.apache.lucene.codecs.lucene50.Lucene50DocValuesFormat.fieldsProducer(Lucene50DocValuesFormat.java:182)
      	at org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66)
      	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:267)
      	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:357)
      	at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
      	at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:68)
      	at org.apache.lucene.index.SegmentDocValuesProducer.<init>(SegmentDocValuesProducer.java:63)
      	at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:167)
      	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:109)
      	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
      	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
      	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:556)
      	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
      	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
      	at org.apache.lucene.index.TestNumericDocValuesUpdates.testUpdateSegmentWithNoDocValues(TestNumericDocValuesUpdates.java:769)
      

      A one-line change to the existing test (on trunk) causes this corruption:

      Index: lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java
      ===================================================================
      --- lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java	(revision 1639580)
      +++ lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java	(working copy)
      @@ -750,6 +750,7 @@
           // second segment with no NDV
           doc = new Document();
           doc.add(new StringField("id", "doc1", Store.NO));
      +    doc.add(new NumericDocValuesField("foo", 3));
           writer.addDocument(doc);
           doc = new Document();
           doc.add(new StringField("id", "doc2", Store.NO)); // document that isn't updated
      

      For some reason, the base doc values for the 2nd segment is not being written, but clearly should have (to hold field "foo")... I'm not sure why.

      Attachments

        1. LUCENE-6062.patch
          3 kB
          Robert Muir
        2. LUCENE-6062.patch
          11 kB
          Robert Muir

        Activity

          People

            Unassigned Unassigned
            mikemccand Michael McCandless
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: