Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6062

Index corruption from numeric DV updates

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I hit this while working on on LUCENE-6005: when cutting over TestNumericDocValuesUpdates to the new Document2 API, I accidentally enabled additional docValues in the test, and this this:

      There was 1 failure:
      1) testUpdateSegmentWithNoDocValues(org.apache.lucene.index.TestNumericDocValuesUpdates)
      java.io.FileNotFoundException: _1_Asserting_0.dvm in dir=RAMDirectory@259847e5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30981eab
      	at __randomizedtesting.SeedInfo.seed([0:7C88A439A551C47D]:0)
      	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:645)
      	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
      	at org.apache.lucene.codecs.lucene50.Lucene50DocValuesProducer.<init>(Lucene50DocValuesProducer.java:130)
      	at org.apache.lucene.codecs.lucene50.Lucene50DocValuesFormat.fieldsProducer(Lucene50DocValuesFormat.java:182)
      	at org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66)
      	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:267)
      	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:357)
      	at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
      	at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:68)
      	at org.apache.lucene.index.SegmentDocValuesProducer.<init>(SegmentDocValuesProducer.java:63)
      	at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:167)
      	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:109)
      	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
      	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
      	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:556)
      	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
      	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
      	at org.apache.lucene.index.TestNumericDocValuesUpdates.testUpdateSegmentWithNoDocValues(TestNumericDocValuesUpdates.java:769)
      

      A one-line change to the existing test (on trunk) causes this corruption:

      Index: lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java
      ===================================================================
      --- lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java	(revision 1639580)
      +++ lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java	(working copy)
      @@ -750,6 +750,7 @@
           // second segment with no NDV
           doc = new Document();
           doc.add(new StringField("id", "doc1", Store.NO));
      +    doc.add(new NumericDocValuesField("foo", 3));
           writer.addDocument(doc);
           doc = new Document();
           doc.add(new StringField("id", "doc2", Store.NO)); // document that isn't updated
      

      For some reason, the base doc values for the 2nd segment is not being written, but clearly should have (to hold field "foo")... I'm not sure why.

        Attachments

        1. LUCENE-6062.patch
          11 kB
          Robert Muir
        2. LUCENE-6062.patch
          3 kB
          Robert Muir

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mikemccand Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: