Details
-
Bug
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
1.6.1
-
apache sling9, oak 1.6.1
-
Important
Description
We are getting the following deadlock. Please help! (Production environment)
I have already annotated possible locks and synchronized blocks in between:
"TarMK flush [/opt/condat/epet9/sling/repository/segmentstore]": waiting to lock Monitor@0x00007fedfc00cc28 (Object@0x00000004795519a8, a org/apache/jackrabbit/oak/segment/SegmentId), which is held by "oak-lucene-14" "oak-lucene-14": waiting for ownable synchronizer 0x00000003c13818c0, (a java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync), which is held by "TarMK flush [/opt/condat/epet9/sling/repository/segmentstore]" Thread 28883: (state = BLOCKED) - org.apache.jackrabbit.oak.segment.SegmentId.getSegment() @bci=12, line=121 (Compiled frame) synchronized (this) - org.apache.jackrabbit.oak.segment.Record.getSegment() @bci=4, line=70 (Compiled frame) - org.apache.jackrabbit.oak.segment.BlockRecord.read(int, byte[], int, int) @bci=49, line=57 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentStream.read(byte[], int, int) @bci=314, line=189 (Compiled frame) - com.google.common.io.ByteStreams.read(java.io.InputStream, byte[], int, int) @bci=43, line=828 (Compiled frame) - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[], int, int) @bci=4, line=695 (Compiled frame) - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[]) @bci=5, line=676 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentStream.getString() @bci=93, line=103 (Compiled frame) - org.apache.jackrabbit.oak.segment.Segment.readString(int) @bci=189, line=524 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(org.apache.jackrabbit.oak.segment.Segment, int) @bci=15, line=212 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentBlob.readBlobId(org.apache.jackrabbit.oak.segment.Segment, int) @bci=37, line=167 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.AbstractFileStore$4.consume(int, org.apache.jackrabbit.oak.segment.RecordType, int) @bci=24, line=354 (Compiled frame) - org.apache.jackrabbit.oak.segment.Segment.forEachRecord(org.apache.jackrabbit.oak.segment.Segment$RecordConsumer) @bci=48, line=716 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.AbstractFileStore.populateTarBinaryReferences(org.apache.jackrabbit.oak.segment.Segment, org.apache.jackrabbit.oak.segment.file.TarWriter) @bci=25, line=349 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(org.apache.jackrabbit.oak.segment.SegmentId, byte[], int, int) @bci=136, line=657 (Compiled frame) fileStoreLock.writeLock().lock(); Zeile 639 bis: populateTarBinaryReferences - org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush() @bci=383, line=383 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.flush() @bci=165, line=148 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentWriter.flush() @bci=4, line=143 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.FileStore$7.call() @bci=7, line=373 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.FileStore$7.call() @bci=1, line=370 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.TarRevisions.doFlush(java.util.concurrent.Callable) @bci=25, line=224 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.TarRevisions.flush(java.util.concurrent.Callable) @bci=42, line=212 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.FileStore.flush() @bci=20, line=370 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.FileStore$2.run() @bci=15, line=233 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.SafeRunnable.run() @bci=21, line=67 (Compiled frame) - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame) - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled frame) - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=180 (Compiled frame) - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=294 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame) Locked ownable synchronizers: - <0x00000003c1361228>, (a java/util/concurrent/locks/ReentrantLock$NonfairSync) - <0x00000003c13818c0>, (a java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync) - <0x00000003c1c3a0d8>, (a java/util/concurrent/ThreadPoolExecutor$Worker) Thread 31035: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int) @bci=83, line=967 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int) @bci=10, line=1283 (Compiled frame) - java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5, line=727 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.FileStore$8.call() @bci=158, line=567 (Compiled frame) fileStoreLock.readLock().lock(); - org.apache.jackrabbit.oak.segment.file.FileStore$8.call() @bci=1, line=542 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(org.apache.jackrabbit.oak.segment.SegmentId, java.util.concurrent.Callable) @bci=1, line=95 (Compiled frame) - org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(org.apache.jackrabbit.oak.segment.SegmentId) @bci=14, line=542 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentId.getSegment() @bci=38, line=125 (Compiled frame) synchronized (this) - org.apache.jackrabbit.oak.segment.Record.getSegment() @bci=4, line=70 (Compiled frame) - org.apache.jackrabbit.oak.segment.BlockRecord.read(int, byte[], int, int) @bci=49, line=57 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentStream.read(byte[], int, int) @bci=314, line=189 (Compiled frame) - com.google.common.io.ByteStreams.read(java.io.InputStream, byte[], int, int) @bci=64, line=833 (Compiled frame) - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[], int, int) @bci=4, line=695 (Compiled frame) - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[]) @bci=5, line=676 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentStream.getString() @bci=93, line=103 (Compiled frame) - org.apache.jackrabbit.oak.segment.Segment.readString(int) @bci=189, line=524 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(org.apache.jackrabbit.oak.segment.Segment, int) @bci=15, line=212 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentBlob.length() @bci=124, line=115 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.<init>(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeBuilder, java.lang.String, org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$BlobFactory) @bci=204, line=409 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.<init>(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeBuilder, java.lang.String, org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$BlobFactory) @bci=25, line=589 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory.fileLength(java.lang.String) @bci=64, line=176 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.copyFilesToLocal(org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory$CORFileReference, boolean, boolean) @bci=195, line=214 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.prefetchIndexFiles() @bci=96, line=170 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.<init>(org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier, org.apache.lucene.store.Directory, org.apache.lucene.store.Directory, boolean, java.lang.String, java.util.concurrent.Executor) @bci=85, line=81 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(java.lang.String, org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition, org.apache.lucene.store.Directory, java.lang.String) @bci=35, line=122 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReader(org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition, org.apache.jackrabbit.oak.spi.state.NodeState, java.lang.String, java.lang.String, java.lang.String) @bci=61, line=102 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReaders(org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition, org.apache.jackrabbit.oak.spi.state.NodeState, java.lang.String) @bci=20, line=61 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.plugins.index.lucene.reader.LuceneIndexReaderFactory, org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndexFactory) @bci=17, line=68 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState) @bci=30, line=132 (Compiled frame) - org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState) @bci=66, line=153 (Compiled frame) - org.apache.jackrabbit.oak.segment.MapRecord.compare(org.apache.jackrabbit.oak.segment.MapRecord, org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=197, line=415 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=909, line=608 (Compiled frame) - org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState) @bci=43, line=148 (Compiled frame) - org.apache.jackrabbit.oak.segment.MapRecord.compare(org.apache.jackrabbit.oak.segment.MapRecord, org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=400, line=457 (Compiled frame) - org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=909, line=608 (Compiled frame) - org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(org.apache.jackrabbit.oak.spi.commit.Editor, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState) @bci=34, line=52 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.diffAndUpdate(org.apache.jackrabbit.oak.spi.state.NodeState) @bci=140, line=142 (Compiled frame) - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(org.apache.jackrabbit.oak.spi.state.NodeState) @bci=36, line=113 (Compiled frame) - org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call() @bci=79, line=135 (Compiled frame) - org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call() @bci=1, line=128 (Compiled frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame) - java.lang.Thread.run() @bci=11, line=748 (Compiled frame) Locked ownable synchronizers: - <0x0000000476194f30>, (a java/util/concurrent/ThreadPoolExecutor$Worker)