Uploaded image for project: 'HiveMind'
  1. HiveMind
  2. HIVEMIND-162

Performance bottleneck with threaded services

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.1
    • 1.1.1
    • framework
    • None
    • Linux 2.6.14.2, JBoss 4.0.3, Tomcat 5.5
      Quad Xeon 3.2GHz, 1GB RAM

    Description

      Note: This may be better classified as a Hivemind issue, but it's affecting Tapestry throughput so I'm putting it here.

      We've been running some perf tests using the Grinder (with 20 threads) generating as much load as possible on a single non-trivial page. The page doesn't touch many of our biz logic services, but does have some complex componentry to render.

      We were seeing performance ramp just fine until we reached about 200TPS, using only 50% CPU. No matter how many clients we threw at it, we couldn't get it any higher. A thread dump showed that most threads were bottlenecked on a synchronized method in HiveMind (servicemodel.ThreadedServiceModel.constructServiceForCurrentThread, see Stack #1 below). This was the construction of the threaded ClientPropertyPersistenceStrategy service. Since we don't use the client strategy, I did a little hivemodule.xml magic and got rid of it from the PersistenceStrategy configuration. This gave us a huge increase in throughput, up to 490TPS still using only about 50% CPU, but we still bottlenecked. Again, a threaddump showed the culprit as constructServiceForCurrentThread, this time in the storage of the RequestGlobals (see Stack #2). We can't remove this service, so we've hit a ceiling.

      Until either Tapestry changes its usage of threaded services, or Hivemind is changed to not synchronize in this way, this looks like a ceiling for Tapestry performance. It'd be nice to be able to use all of the CPU on the box

      ================== STACK #1 ==================
      [java] "http-0.0.0.0-8080-7" daemon prio=1 tid=0x6eaf62e0 nid=0x1261 waiting for monitor entry [0x6f69f000..0x6f6a0
      840]
      [java] at org.apache.hivemind.impl.servicemodel.ThreadedServiceModel.constructServiceForCurrentThread(ThreadedS
      erviceModel.java:166)
      [java] - waiting to lock <0x4e861d88> (a org.apache.hivemind.impl.servicemodel.ThreadedServiceModel)
      [java] at org.apache.hivemind.impl.servicemodel.ThreadedServiceModel.getServiceImplementationForCurrentThread(T
      hreadedServiceModel.java:157)
      [java] at $PropertyPersistenceStrategy_107fc3dfcff._service($PropertyPersistenceStrategy_107fc3dfcff.java)
      [java] at $PropertyPersistenceStrategy_107fc3dfcff.getStoredChanges($PropertyPersistenceStrategy_107fc3dfcff.ja
      va)
      [java] at $PropertyPersistenceStrategy_107fc3dfd00.getStoredChanges($PropertyPersistenceStrategy_107fc3dfd00.ja
      va)
      [java] at org.apache.tapestry.record.PropertyPersistenceStrategySourceImpl.getAllStoredChanges(PropertyPersiste
      nceStrategySourceImpl.java:73)
      [java] at $PropertyPersistenceStrategySource_107fc3dfc3c.getAllStoredChanges($PropertyPersistenceStrategySource
      _107fc3dfc3c.java)
      [java] at org.apache.tapestry.record.PageRecorderImpl.getChanges(PageRecorderImpl.java:68)
      [java] at org.apache.tapestry.record.PageRecorderImpl.rollback(PageRecorderImpl.java:73)
      [java] at org.apache.tapestry.engine.RequestCycle.loadPage(RequestCycle.java:277)
      [java] at org.apache.tapestry.engine.RequestCycle.getPage(RequestCycle.java:249)
      [java] at org.apache.tapestry.engine.RequestCycle.activate(RequestCycle.java:612)
      [java] at org.apache.tapestry.engine.PageService.service(PageService.java:66)
      [java] at $IEngineService_107fc3dfc4c.service($IEngineService_107fc3dfc4c.java)
      [java] at org.apache.tapestry.services.impl.EngineServiceOuterProxy.service(EngineServiceOuterProxy.java:65)
      [java] at org.apache.tapestry.engine.AbstractEngine.service(AbstractEngine.java:248)
      [java] at org.apache.tapestry.services.impl.InvokeEngineTerminator.service(InvokeEngineTerminator.java:60)
      [java] at $WebRequestServicer_107fc3dfc28.service($WebRequestServicer_107fc3dfc28.java)
      [java] at $WebRequestServicer_107fc3dfc24.service($WebRequestServicer_107fc3dfc24.java)
      [java] at org.apache.tapestry.services.impl.WebRequestServicerPipelineBridge.service(WebRequestServicerPipeline
      Bridge.java:56)
      [java] etc...

      ================== STACK #2 ==================
      [java] "http-0.0.0.0-8080-52" daemon prio=1 tid=0x89613328 nid=0x20a9 waiting for monitor entry [0x86d7e000..0x86d7
      f740]
      [java] at org.apache.hivemind.impl.servicemodel.ThreadedServiceModel.constructServiceForCurrentThread(ThreadedS
      erviceModel.java:166)
      [java] - waiting to lock <0x9503a218> (a org.apache.hivemind.impl.servicemodel.ThreadedServiceModel)
      [java] at org.apache.hivemind.impl.servicemodel.ThreadedServiceModel.getServiceImplementationForCurrentThread(T
      hreadedServiceModel.java:157)
      [java] at $RequestGlobals_1080261890e._service($RequestGlobals_1080261890e.java)
      [java] at $RequestGlobals_1080261890e.store($RequestGlobals_1080261890e.java)
      [java] at $RequestGlobals_1080261890f.store($RequestGlobals_1080261890f.java)
      [java] at org.apache.tapestry.services.impl.WebRequestServicerPipelineBridge.service(WebRequestServicerPipeline
      Bridge.java:49)
      [java] at $ServletRequestServicer_108026188f4.service($ServletRequestServicer_108026188f4.java)
      [java] at org.apache.tapestry.request.DecodedRequestInjector.service(DecodedRequestInjector.java:55)
      [java] at $ServletRequestServicerFilter_108026188f0.service($ServletRequestServicerFilter_108026188f0.java)
      [java] at $ServletRequestServicer_108026188f6.service($ServletRequestServicer_108026188f6.java)
      [java] at org.apache.tapestry.multipart.MultipartDecoderFilter.service(MultipartDecoderFilter.java:52)
      [java] at $ServletRequestServicerFilter_108026188ee.service($ServletRequestServicerFilter_108026188ee.java)
      [java] at $ServletRequestServicer_108026188f6.service($ServletRequestServicer_108026188f6.java)
      [java] at org.apache.tapestry.services.impl.SetupRequestEncoding.service(SetupRequestEncoding.java:53)
      [java] at $ServletRequestServicerFilter_108026188f2.service($ServletRequestServicerFilter_108026188f2.java)
      [java] at $ServletRequestServicer_108026188f6.service($ServletRequestServicer_108026188f6.java)
      [java] at $ServletRequestServicer_108026188e8.service($ServletRequestServicer_108026188e8.java)
      [java] at org.apache.tapestry.ApplicationServlet.doService(ApplicationServlet.java:123)
      [java] at com.zillow.web.ZillowServlet.doService(ZillowServlet.java:35)
      [java] at org.apache.tapestry.ApplicationServlet.doGet(ApplicationServlet.java:79)
      [java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
      [java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      [java] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      [java] etc...

      Attachments

        1. hivemind-1.2-alpha-1.jar
          379 kB
          Howard Lewis Ship

        Activity

          People

            hlship Howard Lewis Ship
            jefflub Jeff Lubetkin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment