[LUCENE-9715] EOF error in VectorValues in Lucene nightly benchmarks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Component/s: core/search

Environment

OS: Arch Linux

Java versions:

openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+8)
OpenJDK 64-Bit Server VM (build 11.0.10+8, mixed mode)

 

• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Anton Hägerstrand
• Created: 31/Jan/21 19:55
• Updated: 28/Aug/22 16:13
• Resolved: 08/Feb/21 13:39

Description

Hi! When running the nightly benchmarks, I can consistently reproduce an EOF exception in the VectorValues code:

TASK LEN=150000
Task repeat count 1000
Tasks file /home/anton/dev/lucene-bench-home/util/tasks/wikinightly.tasks
Num task per cat 5
EXC: <vector:knn:<golf>[-0.07267512,...]>
Exception in thread "Thread-2" java.lang.RuntimeException: java.lang.RuntimeException: java.io.EOFException: seek past EOF: MMapIndexInput(path="/home/anton/dev/lucene-bench-home/indices/lucene_bench_2021-01-25_f670131_medium_1thread/index/_32.vec") [slice=vector-data]
	at perf.TaskThreads$TaskThread.run(TaskThreads.java:105)
Caused by: java.lang.RuntimeException: java.io.EOFException: seek past EOF: MMapIndexInput(path="/home/anton/dev/lucene-bench-home/indices/lucene_bench_2021-01-25_f670131_medium_1thread/index/_32.vec") [slice=vector-data]
	at perf.SearchTask.go(SearchTask.java:322)
	at perf.TaskThreads$TaskThread.run(TaskThreads.java:91)
Caused by: java.io.EOFException: seek past EOF: MMapIndexInput(path="/home/anton/dev/lucene-bench-home/indices/lucene_bench_2021-01-25_f670131_medium_1thread/index/_32.vec") [slice=vector-data]
	at org.apache.lucene.store.ByteBufferIndexInput.seek(ByteBufferIndexInput.java:255)
	at org.apache.lucene.store.ByteBufferIndexInput$MultiBufferImpl.seek(ByteBufferIndexInput.java:575)
	at org.apache.lucene.codecs.lucene90.Lucene90VectorReader$OffHeapVectorValues.vectorValue(Lucene90VectorReader.java:432)
	at org.apache.lucene.util.hnsw.HnswGraph.search(HnswGraph.java:118)
	at org.apache.lucene.codecs.lucene90.Lucene90VectorReader$OffHeapVectorValues.search(Lucene90VectorReader.java:409)
	at perf.KnnQuery$KnnWeight.scorer(KnnQuery.java:88)
	at org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:743)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:664)
	at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:510)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:520)
	at perf.SearchTask.go(SearchTask.java:263)
	... 1 more
EXC: <vector:knn:<many geografia>[0.02625591,...]>

I have tried this both on eb24e95731b9f865b95b821c1745264fdc58119 which was head of master/trunk about a week ago, as well as on f670131cbccf42fdde378ee47f9b01977ebbd147 from https://github.com/apache/lucene-solr/pull/2239.

Command was, in the lucene benchmark setup:

      run: java -server -Xms8g -Xmx8g -XX:+FlightRecorder -XX:StartFlightRecording=name=Default,filename=/home/anton/dev/lucene-bench-home/jfr/lucene_bench_2021-01-31_f670131_medium_1thread_search.jfr,settings=profile -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -classpath /home/anton/dev/lucene-bench-home/trunk/lucene/core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar:/home/anton/dev/lucene-bench-home/trunk/lucene/core/build/classes/java/test:/home/anton/dev/lucene-bench-home/trunk/lucene/sandbox/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/misc/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/facet/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/analysis/common/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/analysis/icu/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/queryparser/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/grouping/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/suggest/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/highlighter/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/codecs/build/classes/java/main:/home/anton/dev/lucene-bench-home/trunk/lucene/queries/build/classes/java/main:/home/anton/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.8.2/ccb3ef933ead6b5d766fa571582ddb9b447e48c4/hppc-0.8.2.jar:/home/anton/dev/lucene-bench-home/util/lib/HdrHistogram.jar:/home/anton/dev/lucene-bench-home/util/build perf.SearchPerfTest -dirImpl MMapDirectory -indexPath /home/anton/dev/lucene-bench-home/indices/lucene_bench_2021-01-31_f670131_medium_1thread -facets taxonomy:Date;Date -facets taxonomy:Month;Month -facets taxonomy:DayOfYear;DayOfYear -facets sortedset:Month;Month -facets sortedset:DayOfYear;DayOfYear -analyzer StandardAnalyzerNoStopWords -taskSource /home/anton/dev/lucene-bench-home/util/tasks/wikinightly.tasks -searchThreadCount 2 -taskRepeatCount 1000 -field body -tasksPerCat 5 -staticSeed -8035476 -seed -8826252 -similarity BM25Similarity -commit multi -hiliteImpl FastVectorHighlighter -log /home/anton/dev/lucene-bench-home/logs/jfrtest.jfrtest.0 -topN 10 -printHeap -pk -vectorDict /home/anton/dev/lucene-bench-home/data/glove.6B.100d.txt

Which should be the same that the nighly benchmarks run.

Issue Links

• links to: GitHub Pull Request #2303