Description
Sometimes, when initializing an Oak Lucene index, a class loading deadlock can occur. Unfortunately, no deadlock is reported by the JVM when creating a full thread dump, but the thread dump typically shows the threads below. The root cause seems to be LUCENE-6482, and the reason for that is described in http://ternarysearch.blogspot.it/2013/07/static-initialization-deadlock.html
I have created a simple, reproducible test case, and found a simple workaround in Oak, which is to load the OakCodec before a custom codec. This ensures the class OakCodec, and all superclasses, are loaded before the static initializer of Codec is run. Test case see below (un-commenting the commented line will make it work, otherwise the test results in a deadlock most of the time).
java.lang.Thread.State: RUNNABLE at org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition.createCodec(IndexDefinition.java:1301) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition.<init>(IndexDefinition.java:260) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition.<init>(IndexDefinition.java:228) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:48) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:179) - locked <0x00000007ff915448> (a org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:154) at org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:250) at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:1016) at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:949) at org.apache.jackrabbit.oak.query.ast.SelectorImpl.prepare(SelectorImpl.java:288) at org.apache.jackrabbit.oak.query.QueryImpl.prepare(QueryImpl.java:631) at org.apache.jackrabbit.oak.query.QueryEngineImpl.prepareAndSelect(QueryEngineImpl.java:298) at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:273) at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:233) at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:314) at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:308) at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:304) at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.getTree(IdentifierManager.java:133) at org.apache.jackrabbit.oak.security.authentication.token.TokenProviderImpl.getTokenInfo(TokenProviderImpl.java:250) at org.apache.jackrabbit.oak.security.authentication.token.TokenAuthentication.validateCredentials(TokenAuthentication.java:81) "aysnc-index-update-fulltext-async" prio=5 tid=0x00007fe845e51800 nid=0xb407 in Object.wait() [0x0000700005f2f000] java.lang.Thread.State: RUNNABLE at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:379) at org.apache.lucene.util.NamedSPILoader.reload(NamedSPILoader.java:67) - locked <0x00000007d2ba9120> (a org.apache.lucene.util.NamedSPILoader) at org.apache.lucene.util.NamedSPILoader.<init>(NamedSPILoader.java:45) at org.apache.lucene.util.NamedSPILoader.<init>(NamedSPILoader.java:37) at org.apache.lucene.codecs.Codec.<clinit>(Codec.java:41) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition.createCodec(IndexDefinition.java:1299) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition.<init>(IndexDefinition.java:260) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition.<init>(IndexDefinition.java:224) at org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.<init>(LuceneIndexEditorContext.java:170) at org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.<init>(LuceneIndexEditor.java:132) at org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorProvider.getIndexEditor(LuceneIndexEditorProvider.java:72) at org.apache.jackrabbit.oak.plugins.index.CompositeIndexEditorProvider.getIndexEditor(CompositeIndexEditorProvider.java:74) at org.apache.jackrabbit.oak.spi.whiteboard.WhiteboardIndexEditorProvider.getIndexEditor(WhiteboardIndexEditorProvider.java:52) at org.apache.jackrabbit.oak.plugins.index.IndexUpdate.collectIndexEditors(IndexUpdate.java:201) at org.apache.jackrabbit.oak.plugins.index.IndexUpdate.enter(IndexUpdate.java:144) at org.apache.jackrabbit.oak.spi.commit.VisibleEditor.enter(VisibleEditor.java:57) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:49) at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:510) at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:439)
Test case:
public class StaticInitDeadlock {
public static class Codec {
static {
debug("<clinit>");
forName("OtherCodec");
forName("OakCodec");
debug("<clinit> done");
}
static Codec forName(String name) {
debug("Codec.forName(" + name + ")");
try {
return (Codec) Class.forName("StaticInitDeadlock$" + name)
.newInstance();
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
}
static class OtherCodec extends Codec {
static {
debug("OtherCodec <clinit>");
}
}
static class OakCodec extends Codec {
static {
debug("OakCodec <clinit>");
}
}
static void createCodec(String codecName) {
debug("createCodec (" + codecName + ")");
if (codecName != null) {
// new OakCodec();
Codec.forName(codecName);
} else {
new OakCodec();
}
}
public static void main(String[] args) throws InterruptedException {
Thread thread = new Thread("Thread 2") {
@Override
public void run() {
debug("loading other codec...");
createCodec("OtherCodec");
debug("loading other codec done!");
}
};
thread.start();
Thread.currentThread().setName("Thread 1");
debug("loading default (oak) codec...");
createCodec(null);
debug("joining thread...");
thread.join();
debug("done!");
}
static void debug(String msg) {
Thread t = Thread.currentThread();
StackTraceElement[] st = t.getStackTrace();
int depth = st.length;
String indent = new String(new char[depth]).replace("\0", " ");
System.out.println(t.getName() + ": " + indent + st[2] + " " + msg);
}
}
Output when deadlocked:
Thread 2: StaticInitDeadlock$1.run(StaticInitDeadlock.java:56) loading other codec... Thread 1: StaticInitDeadlock.main(StaticInitDeadlock.java:63) loading default (oak) codec... Thread 2: StaticInitDeadlock.createCodec(StaticInitDeadlock.java:43) createCodec (OtherCodec) Thread 1: StaticInitDeadlock.createCodec(StaticInitDeadlock.java:43) createCodec (null) Thread 2: StaticInitDeadlock$Codec.<clinit>(StaticInitDeadlock.java:11) <clinit> Thread 2: StaticInitDeadlock$Codec.forName(StaticInitDeadlock.java:18) Codec.forName(OtherCodec) Thread 2: StaticInitDeadlock$OtherCodec.<clinit>(StaticInitDeadlock.java:32) OtherCodec <clinit> Thread 2: StaticInitDeadlock$Codec.forName(StaticInitDeadlock.java:18) Codec.forName(OakCodec)
Output without deadlock:
Thread 1: StaticInitDeadlock.main(StaticInitDeadlock.java:63) loading default (oak) codec... Thread 2: StaticInitDeadlock$1.run(StaticInitDeadlock.java:56) loading other codec... Thread 1: StaticInitDeadlock.createCodec(StaticInitDeadlock.java:43) createCodec (null) Thread 2: StaticInitDeadlock.createCodec(StaticInitDeadlock.java:43) createCodec (OtherCodec) Thread 1: StaticInitDeadlock$Codec.<clinit>(StaticInitDeadlock.java:11) <clinit> Thread 1: StaticInitDeadlock$Codec.forName(StaticInitDeadlock.java:18) Codec.forName(OtherCodec) Thread 1: StaticInitDeadlock$OtherCodec.<clinit>(StaticInitDeadlock.java:32) OtherCodec <clinit> Thread 1: StaticInitDeadlock$Codec.forName(StaticInitDeadlock.java:18) Codec.forName(OakCodec) Thread 1: StaticInitDeadlock$Codec.<clinit>(StaticInitDeadlock.java:14) <clinit> done Thread 1: StaticInitDeadlock$OakCodec.<clinit>(StaticInitDeadlock.java:38) OakCodec <clinit> Thread 1: StaticInitDeadlock.main(StaticInitDeadlock.java:65) joining thread... Thread 2: StaticInitDeadlock$Codec.forName(StaticInitDeadlock.java:18) Codec.forName(OtherCodec) Thread 2: StaticInitDeadlock$1.run(StaticInitDeadlock.java:58) loading other codec done! Thread 1: StaticInitDeadlock.main(StaticInitDeadlock.java:67) done!
Attachments
Issue Links
- breaks
-
OAK-4595 OSGiIT failure LuceneIndexProviderService exception
-
- Closed
-