Tapestry 5
  1. Tapestry 5
  2. TAP5-1929

High contention in method InternalComponentResourcesImpl.postRenderCleanup() and NamedSet.getValues()

    Details

      Description

      From the mailing list:

      we want to rollout a Tapestry app very shortly, but we struggle with
      load testing issues. We are currently load testing on one Tomcat 6.0.32:

      • 500 worker threads, tapestry.production-mode=true
      • Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
      • OpenJDK Runtime Environment (IcedTea6 1.7.10)
        (rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16,
        mixed mode))
        and 2 loadrunner test clients.

      After ramping up the concurrent requests (about 5min) we reach the
      maximum at about 450req/sec and get server busy errors. We see a high
      thread contention on InternalComponentResourcesImpl.postRenderCleanup
      currently with the Loop component, as there 10 Loop on the Index page.
      Is there a workaround possible without removing the Loop component from
      the page to increase the throughput? The thread dumps series looks like
      this: 1 thread locks 0x00000000e3858990 and over 400 threads are
      waiting. This lock is persistent over a thread dumps series. I guess the
      private synchronized Map<String, Object> getRenderVariables(boolean
      create) call hits us.

      "http-9080-188" daemon prio=10 tid=0x000000004d463000 nid=0x382a
      runnable [0x0000000055b2f000]
      java.lang.Thread.State: RUNNABLE
      at
      org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206)
      at
      org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302)
      at
      org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136)
      at
      org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133)
      at
      org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171)

      • locked <0x00000000e3858990> (a
        org.apache.tapestry5.internal.util.NamedSet)
        at
        org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546)
      • locked <0x00000000e385c038> (a
        org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl)
        at
        org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522)
        at
        org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443)
        at
        org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
        at
        org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
        at $PageRenderQueue_135675e1f6b934.render(Unknown Source)
        at $PageRenderQueue_135675e1f6b933.render(Unknown Source)
        at
        org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37)
        ...
        "http-9080-499" daemon prio=10 tid=0x000000004dffb000 nid=0x3b7d waiting
        for monitor entry [0x0000000069063000]
        java.lang.Thread.State: BLOCKED (on object monitor)
        at
        org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getRenderVariables(InternalComponentResourcesImpl.java:476)
      • waiting to lock <0x00000000e385c038> (a
        org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl)
        at
        org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:517)
        at
        org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443)
        at
        org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
        at
        org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
        at $PageRenderQueue_135675e1f6b934.render(Unknown Source)
        at $PageRenderQueue_135675e1f6b933.render(Unknown Source)
        at
        org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37)
        ...
        Additionally we experienced a similar issue when using a component with
        a mixin annotation in a loop, that was rendered more than 20 times on
        the page.
        The contention here was on the
        org.apache.tapestry5.internal.util.NamedSet.getValues call

      "http-9080-79" daemon prio=10 tid=0x00000000588dc000 nid=0x3e61 waiting
      for monitor entry [0x000000004ad98000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at
      org.apache.tapestry5.internal.util.NamedSet.getValues(NamedSet.java:78)

      • waiting to lock <0x00000000e2fa4d70> (a
        org.apache.tapestry5.internal.util.NamedSet)
        at
        org.apache.tapestry5.internal.util.NamedSet.getValues(NamedSet.java:257)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl.mixinForClassName(ComponentPageElementImpl.java:892)
        at
        org.apache.tapestry5.internal.structure.ComponentPageElementImpl.getMixinByClassName(ComponentPageElementImpl.java:879)mvn
        at
        org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getMixinByClassName(InternalComponentResourcesImpl.java:366)
        at
        org.apache.tapestry5.internal.transform.MixinWorker$1$1.get(MixinWorker.java:92)
        at
        biz.toc.buyme.client.webapp.core.components.DHTMLTimer.conduit_get__informalElement(DHTMLTimer.java)

      Since these are areas of high contention, we can change them to use an explicit Lock instance so that in the normal case (all readers, no writers), there is no contention or synchronization.

        Activity

        Hide
        Howard M. Lewis Ship added a comment -

        There's some odd race condition going on somewhere ... I think it's caused by the difference between a NamedSet and a CaseInsensitiveMap, but I'm not sure. It's making some tests crash; I don't think I'll have this together before monday.

        Show
        Howard M. Lewis Ship added a comment - There's some odd race condition going on somewhere ... I think it's caused by the difference between a NamedSet and a CaseInsensitiveMap, but I'm not sure. It's making some tests crash; I don't think I'll have this together before monday.
        Hide
        Howard M. Lewis Ship added a comment -

        Now what I'm seeing is that ParameterWorker is writing code to execute in a specific order (so that "value" parameter is fully bound before "encoder" is, since value is a primary, and encoder is dependent) and that's not always happening. I'm not sure what other change I've made has triggered this.

        Show
        Howard M. Lewis Ship added a comment - Now what I'm seeing is that ParameterWorker is writing code to execute in a specific order (so that "value" parameter is fully bound before "encoder" is, since value is a primary, and encoder is dependent) and that's not always happening. I'm not sure what other change I've made has triggered this.
        Hide
        Howard M. Lewis Ship added a comment -

        Think I have humpty dumpty back together again, and better than ever. Fortunately, pretty much all of it was internal (and rarely seen) services and interfaces.

        Show
        Howard M. Lewis Ship added a comment - Think I have humpty dumpty back together again, and better than ever. Fortunately, pretty much all of it was internal (and rarely seen) services and interfaces.
        Hide
        Hudson added a comment -

        Integrated in tapestry-5.3-freestyle #27 (See https://builds.apache.org/job/tapestry-5.3-freestyle/27/)
        TAP5-1929: Performance improvements
        Rework page loading and lazily initialization inside InternalComponentResourcesImpl to avoid some synchronization hotspots (Revision 1337454)

        Result = FAILURE
        hlship : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337454
        Files :

        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/ComponentResources.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/ComponentPageElementImpl.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/InternalComponentResourcesImpl.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/Page.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/PageImpl.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/PageResetListener.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/StructureMessages.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/transform/InjectComponentWorker.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/transform/ParameterConduit.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/transform/ParameterWorker.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/util/NamedSet.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/runtime/PageLifecycleAdapter.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/runtime/PageLifecycleCallbackHub.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/runtime/PageLifecycleListener.java
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/resources/org/apache/tapestry5/internal/structure/StructureStrings.properties
        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/test/java/org/apache/tapestry5/internal/structure/PageImplTest.java
        Show
        Hudson added a comment - Integrated in tapestry-5.3-freestyle #27 (See https://builds.apache.org/job/tapestry-5.3-freestyle/27/ ) TAP5-1929 : Performance improvements Rework page loading and lazily initialization inside InternalComponentResourcesImpl to avoid some synchronization hotspots (Revision 1337454) Result = FAILURE hlship : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337454 Files : /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/ComponentResources.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/ComponentPageElementImpl.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/InternalComponentResourcesImpl.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/Page.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/PageImpl.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/PageResetListener.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/StructureMessages.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/transform/InjectComponentWorker.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/transform/ParameterConduit.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/transform/ParameterWorker.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/util/NamedSet.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/runtime/PageLifecycleAdapter.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/runtime/PageLifecycleCallbackHub.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/runtime/PageLifecycleListener.java /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/resources/org/apache/tapestry5/internal/structure/StructureStrings.properties /tapestry/tapestry5/branches/5.3/tapestry-core/src/test/java/org/apache/tapestry5/internal/structure/PageImplTest.java
        Hide
        Robert Lentz added a comment - - edited

        We rerun the loadtest with tapestry-5.3-freestyle (Last Changed Rev: 1337454) and the high thread contention shifted to the InternalComponentResourcesImpl.resetParameterConduits() call

        "http-9080-300" daemon prio=10 tid=0x00002aaac0e57000 nid=0x6e47 runnable [0x0000000040ce3000]
        java.lang.Thread.State: RUNNABLE
        at org.apache.tapestry5.ioc.internal.util.CollectionFactory.newSet(CollectionFactory.java:55)
        at org.apache.tapestry5.internal.util.NamedSet.getValues(NamedSet.java:91)
        at org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:202)
        at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:651)

        • locked <0x00000000e3ad4280> (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl)
          at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:629)
          at org.apache.tapestry5.corelib.base.AbstractLink.postRenderCleanup(AbstractLink.java)
          at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85)
          at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:970)
          at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61)
          at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443)
          at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
          at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
          at $PageRenderQueue_144f2bffe2549b.render(Unknown Source)
          at $PageRenderQueue_144f2bffe2549a.render(Unknown Source)
          at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37)

        and 400 threads waiting on the same lock or on other different InternalComponentResourcesImpl blocking 100's of threads.

        "http-9080-79" daemon prio=10 tid=0x00002aaac1806000 nid=0x69f4 waiting for monitor entry [0x000000004ae44000]
        java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:649)

        • waiting to lock <0x00000000e3ad4280> (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl)
          at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:629)
          at org.apache.tapestry5.corelib.base.AbstractLink.postRenderCleanup(AbstractLink.java)
          at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85)
          at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:970)
          at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61)
          at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443)
          at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
          at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
          at $PageRenderQueue_144f2bffe2549b.render(Unknown Source)
          at $PageRenderQueue_144f2bffe2549a.render(Unknown Source)
          at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37)
          at org.apache.tapestry5.services.TapestryModule$30.renderMarkup(TapestryModule.java:1979)
          at $MarkupRenderer_144f2bffe254a0.renderMarkup(Unknown Source)
        Show
        Robert Lentz added a comment - - edited We rerun the loadtest with tapestry-5.3-freestyle (Last Changed Rev: 1337454) and the high thread contention shifted to the InternalComponentResourcesImpl.resetParameterConduits() call "http-9080-300" daemon prio=10 tid=0x00002aaac0e57000 nid=0x6e47 runnable [0x0000000040ce3000] java.lang.Thread.State: RUNNABLE at org.apache.tapestry5.ioc.internal.util.CollectionFactory.newSet(CollectionFactory.java:55) at org.apache.tapestry5.internal.util.NamedSet.getValues(NamedSet.java:91) at org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:202) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:651) locked <0x00000000e3ad4280> (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:629) at org.apache.tapestry5.corelib.base.AbstractLink.postRenderCleanup(AbstractLink.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:970) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_144f2bffe2549b.render(Unknown Source) at $PageRenderQueue_144f2bffe2549a.render(Unknown Source) at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37) and 400 threads waiting on the same lock or on other different InternalComponentResourcesImpl blocking 100's of threads. "http-9080-79" daemon prio=10 tid=0x00002aaac1806000 nid=0x69f4 waiting for monitor entry [0x000000004ae44000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:649) waiting to lock <0x00000000e3ad4280> (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:629) at org.apache.tapestry5.corelib.base.AbstractLink.postRenderCleanup(AbstractLink.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:970) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_144f2bffe2549b.render(Unknown Source) at $PageRenderQueue_144f2bffe2549a.render(Unknown Source) at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37) at org.apache.tapestry5.services.TapestryModule$30.renderMarkup(TapestryModule.java:1979) at $MarkupRenderer_144f2bffe254a0.renderMarkup(Unknown Source)
        Hide
        Hudson added a comment -

        Integrated in tapestry-5.3-freestyle #28 (See https://builds.apache.org/job/tapestry-5.3-freestyle/28/)
        TAP5-1929: Performance improvements
        Extend the lazy initialization reentrant lock to conver the conduits NamedSet as well (Revision 1338276)

        Result = FAILURE
        hlship : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338276
        Files :

        • /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/InternalComponentResourcesImpl.java
        Show
        Hudson added a comment - Integrated in tapestry-5.3-freestyle #28 (See https://builds.apache.org/job/tapestry-5.3-freestyle/28/ ) TAP5-1929 : Performance improvements Extend the lazy initialization reentrant lock to conver the conduits NamedSet as well (Revision 1338276) Result = FAILURE hlship : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338276 Files : /tapestry/tapestry5/branches/5.3/tapestry-core/src/main/java/org/apache/tapestry5/internal/structure/InternalComponentResourcesImpl.java
        Hide
        Howard M. Lewis Ship added a comment -

        This was fixed across a series of releases, actually.

        Show
        Howard M. Lewis Ship added a comment - This was fixed across a series of releases, actually.

          People

          • Assignee:
            Howard M. Lewis Ship
            Reporter:
            Howard M. Lewis Ship
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development