Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
None
-
None
-
Linux 64-bit, Java 1.6, Tomcat, Grails
Description
We have a Grails application serving hundreds of requests per second and this seems to be the most critical hot spot for us. Under high load, most threads are blocked in the following call stack:
"http-apr-8080"-exec-144
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:34)
org.codehaus.groovy.reflection.ClassInfo.lock(ClassInfo.java:268)
org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:193)
org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:214)
org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:747)
org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:780)
org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:772)
org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.castToBoolean(DefaultTypeTransformation.java:156)
org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.booleanUnbox(DefaultTypeTransformation.java:65)
Grails uses InvokerHelper a lot, which calls ClassInfo.getMetaClass which uses locking. This is stop-the-world lock affecting all threads (they all hit the same ClassInfo instance). Note that 99,999% of time the locking is useless as nothing is modified (typically all metaclasses getting modified on startup).
There are several related tickets: GROOVY-3557 and GROOVY-5059, not really solving the issue.
The solution could be to use more fine-grained locks (ReadWriteLock) or Atomics. Should be easy to implement, but need to isolate modification part from read-only parts.
Doing so can be a good boost to overall Grails performance.