Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
5.3.3, 5.4
Description
From the mailing list:
we want to rollout a Tapestry app very shortly, but we struggle with
load testing issues. We are currently load testing on one Tomcat 6.0.32:
- 500 worker threads, tapestry.production-mode=true
- Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
- OpenJDK Runtime Environment (IcedTea6 1.7.10)
(rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16,
mixed mode))
and 2 loadrunner test clients.
After ramping up the concurrent requests (about 5min) we reach the
maximum at about 450req/sec and get server busy errors. We see a high
thread contention on InternalComponentResourcesImpl.postRenderCleanup
currently with the Loop component, as there 10 Loop on the Index page.
Is there a workaround possible without removing the Loop component from
the page to increase the throughput? The thread dumps series looks like
this: 1 thread locks 0x00000000e3858990 and over 400 threads are
waiting. This lock is persistent over a thread dumps series. I guess the
private synchronized Map<String, Object> getRenderVariables(boolean
create) call hits us.
"http-9080-188" daemon prio=10 tid=0x000000004d463000 nid=0x382a
runnable [0x0000000055b2f000]
java.lang.Thread.State: RUNNABLE
at
org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206)
at
org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133)
at
org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171)
- locked <0x00000000e3858990> (a
org.apache.tapestry5.internal.util.NamedSet)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546) - locked <0x00000000e385c038> (a
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522)
at
org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443)
at
org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
at
org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
at $PageRenderQueue_135675e1f6b934.render(Unknown Source)
at $PageRenderQueue_135675e1f6b933.render(Unknown Source)
at
org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37)
...
"http-9080-499" daemon prio=10 tid=0x000000004dffb000 nid=0x3b7d waiting
for monitor entry [0x0000000069063000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getRenderVariables(InternalComponentResourcesImpl.java:476) - waiting to lock <0x00000000e385c038> (a
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:517)
at
org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443)
at
org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
at
org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
at $PageRenderQueue_135675e1f6b934.render(Unknown Source)
at $PageRenderQueue_135675e1f6b933.render(Unknown Source)
at
org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37)
...
Additionally we experienced a similar issue when using a component with
a mixin annotation in a loop, that was rendered more than 20 times on
the page.
The contention here was on the
org.apache.tapestry5.internal.util.NamedSet.getValues call
"http-9080-79" daemon prio=10 tid=0x00000000588dc000 nid=0x3e61 waiting
for monitor entry [0x000000004ad98000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.tapestry5.internal.util.NamedSet.getValues(NamedSet.java:78)
- waiting to lock <0x00000000e2fa4d70> (a
org.apache.tapestry5.internal.util.NamedSet)
at
org.apache.tapestry5.internal.util.NamedSet.getValues(NamedSet.java:257)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.mixinForClassName(ComponentPageElementImpl.java:892)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.getMixinByClassName(ComponentPageElementImpl.java:879)mvn
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getMixinByClassName(InternalComponentResourcesImpl.java:366)
at
org.apache.tapestry5.internal.transform.MixinWorker$1$1.get(MixinWorker.java:92)
at
biz.toc.buyme.client.webapp.core.components.DHTMLTimer.conduit_get__informalElement(DHTMLTimer.java)
Since these are areas of high contention, we can change them to use an explicit Lock instance so that in the normal case (all readers, no writers), there is no contention or synchronization.