[LUCENE-9467] TestOfflineSorter fails with combination of NRTCaching and ByteBuffersDirectory

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.7, 8.6.1
• Fix Version/s: 8.7
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Simon Willnauer
• Created: 17/Aug/20 09:48
• Updated: 28/Aug/22 16:06
• Resolved: 17/Aug/20 16:34

Description

TestOfflineSorter#testThreadSafety fails on 8.x since NRTCachingDirectory uses the delegates openInput method to slowly check if the file exists. This is not present anymore on main line. The problem is that ByteBuffersDirectory fires an AccessDeniedException when the file is not fully written yet.

00:27:31    [junit4] Suite: org.apache.lucene.util.TestOfflineSorter
00:27:31    [junit4]   2> KMN 14, 2020 6:27:30 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
00:27:31    [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-7318,5,TGRP-TestOfflineSorter]
00:27:31    [junit4]   2> java.lang.RuntimeException: java.nio.file.AccessDeniedException: Can't open a file still open for writing: unsorted_tmp_6.tmp
00:27:31    [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
00:27:31    [junit4]   2> 	at org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
00:27:31    [junit4]   2> Caused by: java.nio.file.AccessDeniedException: Can't open a file still open for writing: unsorted_tmp_6.tmp
00:27:31    [junit4]   2> 	at org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
00:27:31    [junit4]   2> 	at org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
00:27:31    [junit4]   2> 	at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
00:27:31    [junit4]   2> 	at org.apache.lucene.store.NRTCachingDirectory.slowFileExists(NRTCachingDirectory.java:280)
00:27:31    [junit4]   2> 	at org.apache.lucene.store.NRTCachingDirectory.createTempOutput(NRTCachingDirectory.java:254)
00:27:31    [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:707)
00:27:31    [junit4]   2> 	at org.apache.lucene.util.TestOfflineSorter.checkSort(TestOfflineSorter.java:184)
00:27:31    [junit4]   2> 	at org.apache.lucene.util.TestOfflineSorter.access$100(TestOfflineSorter.java:48)
00:27:31    [junit4]   2> 	at org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:268)
00:27:31    [junit4]   2> 
00:27:31    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestOfflineSorter -Dtests.method=testThreadSafety -Dtests.seed=B1129484013118F7 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=cgg-UG -Dtests.timezone=Asia/Dacca -Dtests.asserts=true -Dtests.file.encoding=UTF8
00:27:31    [junit4] ERROR   0.04s J3 | TestOfflineSorter.testThreadSafety <<<
00:27:31    [junit4]    > Throwable #1: java.lang.AssertionError
00:27:31    [junit4]    > 	at org.apache.lucene.util.TestOfflineSorter.testThreadSafety(TestOfflineSorter.java:283)
00:27:31    [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
00:27:31    [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
00:27:31    [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
00:27:31    [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
00:27:31    [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:834)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7993, name=Thread-7318, state=RUNNABLE, group=TGRP-TestOfflineSorter]
00:27:31    [junit4]    > Caused by: java.lang.RuntimeException: java.nio.file.AccessDeniedException: Can't open a file still open for writing: unsorted_tmp_6.tmp
00:27:31    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
00:27:31    [junit4]    > 	at org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
00:27:31    [junit4]    > Caused by: java.nio.file.AccessDeniedException: Can't open a file still open for writing: unsorted_tmp_6.tmp
00:27:31    [junit4]    > 	at org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
00:27:31    [junit4]    > 	at org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
00:27:31    [junit4]    > 	at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
00:27:31    [junit4]    > 	at org.apache.lucene.store.NRTCachingDirectory.slowFileExists(NRTCachingDirectory.java:280)
00:27:31    [junit4]    > 	at org.apache.lucene.store.NRTCachingDirectory.createTempOutput(NRTCachingDirectory.java:254)
00:27:31    [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:707)
00:27:31    [junit4]    > 	at org.apache.lucene.util.TestOfflineSorter.checkSort(TestOfflineSorter.java:184)
00:27:31    [junit4]    > 	at org.apache.lucene.util.TestOfflineSorter.access$100(TestOfflineSorter.java:48)
00:27:31    [junit4]    > 	at org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:268)