[OAK-8729] Lucene Directory concurrency issue

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0, 1.14.0, 1.16.0, 1.18.0
• Fix Version/s: 1.20.0
• Component/s: lucene
• Assignee: Thomas Mueller
• Reporter: Thomas Mueller
• Created: 31/Oct/19 17:02
• Updated: 25/Nov/19 09:28
• Resolved: 07/Nov/19 16:10

Description

There is a concurrency issue in the DefaultDirectoryFactory. It is reproducible sometimes using CopyOnWriteDirectoryTest.copyOnWrite(), if run in a loop (1000 times). The problem is that the MemoryNodeBuilder is used concurrently:

• thread 1 is closing the directory (after writing to it)
• thread 2 is trying to create a new file

	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.exists(MemoryNodeBuilder.java:284)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setProperty(MemoryNodeBuilder.java:525)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.OakDirectory.close(OakDirectory.java:264)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.BufferedOakDirectory.close(BufferedOakDirectory.java:217)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory$2.run(CopyOnReadDirectory.java:305)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.exists(MemoryNodeBuilder.java:284)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:362)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:356)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.child(MemoryNodeBuilder.java:342)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.OakDirectory.createOutput(OakDirectory.java:214)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.BufferedOakDirectory.createOutput(BufferedOakDirectory.java:178)
	at org.apache.lucene.store.Directory.copy(Directory.java:184)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory$3.call(CopyOnWriteDirectory.java:322)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory$3.call(CopyOnWriteDirectory.java:1)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory$2$1.call(CopyOnWriteDirectory.java:105)
	at org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory$2$1.call(CopyOnWriteDirectory.java:1)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• is caused by: OAK-8097 Load Lucene index files before writing to the index