[LUCENE-8790] Spooky exception merging doc values

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 7.5
• Component/s: core/index

Environment

We are on a Lucene 7.x snapshot, githash 

935b0c89c6ecb446d7f05d938207760cd64bcd04, using the default Codec, with a static sort.

• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Michael McCandless
• Created: 02/May/19 23:53
• Updated: 28/Aug/22 15:44

Description

We hit this exciting exception; we don't have a test case reproducing it, and staring at the code, I don't see how we can hit a NullPointerException on this line:

[May 2, 2019, 7:24 PM] Barrowman, Adam: 2019-05-02T18:32:10,561 [ERROR] (Lucene Merge Thread #1) com.amazon.lucene.util.UncaughtExceptionHandler: Uncaught exception: org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException in thread Thread[Lucene Merge Thread #1,5,main] org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
Caused by: java.lang.NullPointerException
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValuesSingleBlock(Lucene70DocValuesConsumer.java:279)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValues(Lucene70DocValuesConsumer.java:263)
at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedNumericField(Lucene70DocValuesConsumer.java:536)
at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:371)
at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:143)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:182)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:126)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4438)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4060)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
at com.amazon.lucene.index.ConcurrentMergeSchedulerWrapper.doMerge(ConcurrentMergeSchedulerWrapper.java:54)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)

It seems like the encode.get(v) somehow returned null, which should not happen as long as the values we iterated from the SortedNumericValues were the same up above (in writeValues) and in writeValuesSingleBlock.  Confused...

Note that we are using a 7.x snapshot, so it is possible this was a bug in 7.x at that time, fixed before the next 7.x release though when I compare the affected code against 8.x backwards codec, it looks the same.