Details
Major Description
Note: This may be better classified as a Hivemind issue, but it's affecting Tapestry throughput so I'm putting it here.
We've been running some perf tests using the Grinder (with 20 threads) generating as much load as possible on a single non-trivial page. The page doesn't touch many of our biz logic services, but does have some complex componentry to render.
We were seeing performance ramp just fine until we reached about 200TPS, using only 50% CPU. No matter how many clients we threw at it, we couldn't get it any higher. A thread dump showed that most threads were bottlenecked on a synchronized method in HiveMind (servicemodel.ThreadedServiceModel.constructServiceForCurrentThread, see Stack #1 below). This was the construction of the threaded ClientPropertyPersistenceStrategy service. Since we don't use the client strategy, I did a little hivemodule.xml magic and got rid of it from the PersistenceStrategy configuration. This gave us a huge increase in throughput, up to 490TPS still using only about 50% CPU, but we still bottlenecked. Again, a threaddump showed the culprit as constructServiceForCurrentThread, this time in the storage of the RequestGlobals (see Stack #2). We can't remove this service, so we've hit a ceiling.
Until either Tapestry changes its usage of threaded services, or Hivemind is changed to not synchronize in this way, this looks like a ceiling for Tapestry performance. It'd be nice to be able to use all of the CPU on the box 
================== STACK #1 ==================
[java] "http-0.0.0.0-8080-7" daemon prio=1 tid=0x6eaf62e0 nid=0x1261 waiting for monitor entry [0x6f69f000..0x6f6a0
840]
[java] at org.apache.hivemind.impl.servicemodel.ThreadedServiceModel.constructServiceForCurrentThread(ThreadedS
erviceModel.java:166)
[java] - waiting to lock <0x4e861d88> (a org.apache.hivemind.impl.servicemodel.ThreadedServiceModel)
[java] at org.apache.hivemind.impl.servicemodel.ThreadedServiceModel.getServiceImplementationForCurrentThread(T
hreadedServiceModel.java:157)
[java] at $PropertyPersistenceStrategy_107fc3dfcff._service($PropertyPersistenceStrategy_107fc3dfcff.java)
[java] at $PropertyPersistenceStrategy_107fc3dfcff.getStoredChanges($PropertyPersistenceStrategy_107fc3dfcff.ja
va)
[java] at $PropertyPersistenceStrategy_107fc3dfd00.getStoredChanges($PropertyPersistenceStrategy_107fc3dfd00.ja
va)
[java] at org.apache.tapestry.record.PropertyPersistenceStrategySourceImpl.getAllStoredChanges(PropertyPersiste
nceStrategySourceImpl.java:73)
[java] at $PropertyPersistenceStrategySource_107fc3dfc3c.getAllStoredChanges($PropertyPersistenceStrategySource
_107fc3dfc3c.java)
[java] at org.apache.tapestry.record.PageRecorderImpl.getChanges(PageRecorderImpl.java:68)
[java] at org.apache.tapestry.record.PageRecorderImpl.rollback(PageRecorderImpl.java:73)
[java] at org.apache.tapestry.engine.RequestCycle.loadPage(RequestCycle.java:277)
[java] at org.apache.tapestry.engine.RequestCycle.getPage(RequestCycle.java:249)
[java] at org.apache.tapestry.engine.RequestCycle.activate(RequestCycle.java:612)
[java] at org.apache.tapestry.engine.PageService.service(PageService.java:66)
[java] at $IEngineService_107fc3dfc4c.service($IEngineService_107fc3dfc4c.java)
[java] at org.apache.tapestry.services.impl.EngineServiceOuterProxy.service(EngineServiceOuterProxy.java:65)
[java] at org.apache.tapestry.engine.AbstractEngine.service(AbstractEngine.java:248)
[java] at org.apache.tapestry.services.impl.InvokeEngineTerminator.service(InvokeEngineTerminator.java:60)
[java] at $WebRequestServicer_107fc3dfc28.service($WebRequestServicer_107fc3dfc28.java)
[java] at $WebRequestServicer_107fc3dfc24.service($WebRequestServicer_107fc3dfc24.java)
[java] at org.apache.tapestry.services.impl.WebRequestServicerPipelineBridge.service(WebRequestServicerPipeline
Bridge.java:56)
[java] etc...
================== STACK #2 ==================
[java] "http-0.0.0.0-8080-52" daemon prio=1 tid=0x89613328 nid=0x20a9 waiting for monitor entry [0x86d7e000..0x86d7
f740]
[java] at org.apache.hivemind.impl.servicemodel.ThreadedServiceModel.constructServiceForCurrentThread(ThreadedS
erviceModel.java:166)
[java] - waiting to lock <0x9503a218> (a org.apache.hivemind.impl.servicemodel.ThreadedServiceModel)
[java] at org.apache.hivemind.impl.servicemodel.ThreadedServiceModel.getServiceImplementationForCurrentThread(T
hreadedServiceModel.java:157)
[java] at $RequestGlobals_1080261890e._service($RequestGlobals_1080261890e.java)
[java] at $RequestGlobals_1080261890e.store($RequestGlobals_1080261890e.java)
[java] at $RequestGlobals_1080261890f.store($RequestGlobals_1080261890f.java)
[java] at org.apache.tapestry.services.impl.WebRequestServicerPipelineBridge.service(WebRequestServicerPipeline
Bridge.java:49)
[java] at $ServletRequestServicer_108026188f4.service($ServletRequestServicer_108026188f4.java)
[java] at org.apache.tapestry.request.DecodedRequestInjector.service(DecodedRequestInjector.java:55)
[java] at $ServletRequestServicerFilter_108026188f0.service($ServletRequestServicerFilter_108026188f0.java)
[java] at $ServletRequestServicer_108026188f6.service($ServletRequestServicer_108026188f6.java)
[java] at org.apache.tapestry.multipart.MultipartDecoderFilter.service(MultipartDecoderFilter.java:52)
[java] at $ServletRequestServicerFilter_108026188ee.service($ServletRequestServicerFilter_108026188ee.java)
[java] at $ServletRequestServicer_108026188f6.service($ServletRequestServicer_108026188f6.java)
[java] at org.apache.tapestry.services.impl.SetupRequestEncoding.service(SetupRequestEncoding.java:53)
[java] at $ServletRequestServicerFilter_108026188f2.service($ServletRequestServicerFilter_108026188f2.java)
[java] at $ServletRequestServicer_108026188f6.service($ServletRequestServicer_108026188f6.java)
[java] at $ServletRequestServicer_108026188e8.service($ServletRequestServicer_108026188e8.java)
[java] at org.apache.tapestry.ApplicationServlet.doService(ApplicationServlet.java:123)
[java] at com.zillow.web.ZillowServlet.doService(ZillowServlet.java:35)
[java] at org.apache.tapestry.ApplicationServlet.doGet(ApplicationServlet.java:79)
[java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
[java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
[java] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
[java] etc...