[LUCENE-6062] Index corruption from numeric DV updates

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.0, 6.0
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Michael McCandless
• Created: 14/Nov/14 08:40
• Updated: 28/Aug/22 14:19
• Resolved: 19/Nov/14 03:24

Description

I hit this while working on on LUCENE-6005: when cutting over TestNumericDocValuesUpdates to the new Document2 API, I accidentally enabled additional docValues in the test, and this this:

There was 1 failure:
1) testUpdateSegmentWithNoDocValues(org.apache.lucene.index.TestNumericDocValuesUpdates)
java.io.FileNotFoundException: _1_Asserting_0.dvm in dir=RAMDirectory@259847e5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30981eab
	at __randomizedtesting.SeedInfo.seed([0:7C88A439A551C47D]:0)
	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:645)
	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
	at org.apache.lucene.codecs.lucene50.Lucene50DocValuesProducer.<init>(Lucene50DocValuesProducer.java:130)
	at org.apache.lucene.codecs.lucene50.Lucene50DocValuesFormat.fieldsProducer(Lucene50DocValuesFormat.java:182)
	at org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66)
	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:267)
	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:357)
	at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
	at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:68)
	at org.apache.lucene.index.SegmentDocValuesProducer.<init>(SegmentDocValuesProducer.java:63)
	at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:167)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:109)
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:556)
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
	at org.apache.lucene.index.TestNumericDocValuesUpdates.testUpdateSegmentWithNoDocValues(TestNumericDocValuesUpdates.java:769)

A one-line change to the existing test (on trunk) causes this corruption:

Index: lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java
===================================================================
--- lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java	(revision 1639580)
+++ lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java	(working copy)
@@ -750,6 +750,7 @@
     // second segment with no NDV
     doc = new Document();
     doc.add(new StringField("id", "doc1", Store.NO));
+    doc.add(new NumericDocValuesField("foo", 3));
     writer.addDocument(doc);
     doc = new Document();
     doc.add(new StringField("id", "doc2", Store.NO)); // document that isn't updated

For some reason, the base doc values for the 2nd segment is not being written, but clearly should have (to hold field "foo")... I'm not sure why.