Tapestry
  1. Tapestry
  2. TAPESTRY-2468

A deadlock related to class loading (and creation) can occur

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 5.0.11
    • Fix Version/s: 5.0.14
    • Component/s: tapestry-ioc
    • Labels:
      None

      Description

      Hi All!

      We encountered a deadlock situation using T5.0.11 under heavy load. Unfortunately we do not have the detailed knowledge about all T5/javassist-secrets to solve this on our own.
      It seems there are two different execution paths resulting in a call to "ClassPool.toClass". In both cases the inner class org.apache.tapestry.internal.services.ComponentInstantiatorSourceImpl$PackageAwareLoader is part of the deadlock.
      On the one hand we have a call to "findClass(ComponentInstantiatorSourceImpl.java:75)" (please see "catalina-40.txt") on the other hand the toClass2 method tries to access this inner class at the same time. But this is only a guess.

      See below both thread dump stacktraces.

      Any hint to solve this problem is highly appreciated.

      Jens

      "catalina-exec-11" Id=67 BLOCKED on org.apache.tapestry.internal.services.ComponentInstantiatorSourceImpl$PackageAwareLoader@adedf6 owned by "catalina-exec-40" Id=96
      at java.lang.ClassLoader.defineClass1(Native Method)
      at java.lang.ClassLoader.defineClass(ClassLoader.java:675)
      at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at javassist.ClassPool.toClass2(ClassPool.java:965)
      at javassist.ClassPool.toClass(ClassPool.java:947)
      at org.apache.tapestry.ioc.internal.services.CtClassSource.createClass(CtClassSource.java:87)
      at org.apache.tapestry.ioc.internal.services.AbstractFab.createClass(AbstractFab.java:106)
      at org.apache.tapestry.internal.services.PropertyConduitSourceImpl.build(PropertyConduitSourceImpl.java:185)
      at org.apache.tapestry.internal.services.PropertyConduitSourceImpl.create(PropertyConduitSourceImpl.java:120)
      at $PropertyConduitSource_11a6cad5e6b.create($PropertyConduitSource_11a6cad5e6b.java)
      at org.apache.tapestry.internal.bindings.PropBindingFactory.newBinding(PropBindingFactory.java:48)
      at $BindingFactory_11a6cad5e6c.newBinding($BindingFactory_11a6cad5e6c.java)
      at $BindingFactory_11a6cad5e63.newBinding($BindingFactory_11a6cad5e63.java)
      at org.apache.tapestry.internal.services.BindingSourceImpl.newBinding(BindingSourceImpl.java:79)
      at $BindingSource_11a6cad5e55.newBinding($BindingSource_11a6cad5e55.java)
      at org.apache.tapestry.internal.services.PageElementFactoryImpl.parseAttributeExpansionExpression(PageElementFactoryImpl.java:127)
      at org.apache.tapestry.internal.services.PageElementFactoryImpl.newBinding(PageElementFactoryImpl.java:299)
      at $PageElementFactory_11a6cad5e4f.newBinding($PageElementFactory_11a6cad5e4f.java)
      at org.apache.tapestry.internal.services.PageLoaderProcessor.findBinding(PageLoaderProcessor.java:264)
      at org.apache.tapestry.internal.services.PageLoaderProcessor.bindParametersFromModel(PageLoaderProcessor.java:217)
      at org.apache.tapestry.internal.services.PageLoaderProcessor.startComponent(PageLoaderProcessor.java:667)
      at org.apache.tapestry.internal.services.PageLoaderProcessor.loadTemplateForComponent(PageLoaderProcessor.java:500)
      at org.apache.tapestry.internal.services.PageLoaderProcessor.workComponentQueue(PageLoaderProcessor.java:807)
      at org.apache.tapestry.internal.services.PageLoaderProcessor.loadPage(PageLoaderProcessor.java:392)
      at org.apache.tapestry.internal.services.PageLoaderImpl.loadPage(PageLoaderImpl.java:59)
      at $PageLoader_11a6cad5e4d.loadPage($PageLoader_11a6cad5e4d.java)
      at org.apache.tapestry.internal.services.PagePoolCache.checkout(PagePoolCache.java:188)
      at org.apache.tapestry.internal.services.PagePoolImpl.checkout(PagePoolImpl.java:107)
      at $PagePool_11a6cad5e4c.checkout($PagePool_11a6cad5e4c.java)
      at org.apache.tapestry.internal.services.RequestPageCacheImpl.get(RequestPageCacheImpl.java:43)
      at $RequestPageCache_11a6cad5e4b.get($RequestPageCache_11a6cad5e4b.java)
      at $RequestPageCache_11a6cad5e0d.get($RequestPageCache_11a6cad5e0d.java)
      at org.apache.tapestry.internal.services.LinkFactoryImpl.createPageLink(LinkFactoryImpl.java:273)
      at $LinkFactory_11a6cad5e47.createPageLink($LinkFactory_11a6cad5e47.java)
      at org.apache.tapestry.internal.structure.PageImpl.createPageLink(PageImpl.java:161)
      at org.apache.tapestry.internal.structure.InternalComponentResourcesImpl.createPageLink(InternalComponentResourcesImpl.java:129)
      at org.apache.tapestry.corelib.components.PageLink.beginRender(PageLink.java:73)
      at org.apache.tapestry.corelib.components.PageLink.beginRender(PageLink.java)
      at org.apache.tapestry.internal.structure.ComponentPageElementImpl$11$1.run(ComponentPageElementImpl.java:338)
      at org.apache.tapestry.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:874)
      at org.apache.tapestry.internal.structure.ComponentPageElementImpl.access$100(ComponentPageElementImpl.java:54)
      at org.apache.tapestry.internal.structure.ComponentPageElementImpl$11.render(ComponentPageElementImpl.java:342)
      at org.apache.tapestry.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:63)
      at org.apache.tapestry.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:84)
      at $PageRenderQueue_11a6cad61e4.render($PageRenderQueue_11a6cad61e4.java)
      at $PageRenderQueue_11a6cad61dd.render($PageRenderQueue_11a6cad61dd.java)
      at org.apache.tapestry.services.TapestryModule$19.renderMarkup(TapestryModule.java:1293)
      at nz.toc.taptoolsapp.services.CustomValidationDecoratorRenderer.renderMarkup(CustomValidationDecoratorRenderer.java:28)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at org.apache.tapestry.services.TapestryModule$23.renderMarkup(TapestryModule.java:1402)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at org.apache.tapestry.services.TapestryModule$22.renderMarkup(TapestryModule.java:1383)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at org.apache.tapestry.services.TapestryModule$21.renderMarkup(TapestryModule.java:1365)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at nz.toc.taptoolsapp.services.AppModule$1.renderMarkup(AppModule.java:79)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at org.apache.tapestry.services.TapestryModule$20.renderMarkup(TapestryModule.java:1347)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at $MarkupRenderer_11a6cad61e2.renderMarkup($MarkupRenderer_11a6cad61e2.java)
      at org.apache.tapestry.internal.services.PageMarkupRendererImpl.renderPageMarkup(PageMarkupRendererImpl.java:55)
      at $PageMarkupRenderer_11a6cad61e0.renderPageMarkup($PageMarkupRenderer_11a6cad61e0.java)
      at org.apache.tapestry.internal.services.PageResponseRendererImpl.renderPageResponse(PageResponseRendererImpl.java:57)
      at $PageResponseRenderer_11a6cad5e45.renderPageResponse($PageResponseRenderer_11a6cad5e45.java)
      at org.apache.tapestry.internal.services.PageRenderRequestHandlerImpl.handle(PageRenderRequestHandlerImpl.java:59)
      at org.apache.tapestry.services.TapestryModule$29.handle(TapestryModule.java:1607)
      at $PageRenderRequestHandler_11a6cad5e46.handle($PageRenderRequestHandler_11a6cad5e46.java)
      at $PageRenderRequestHandler_11a6cad5e3b.handle($PageRenderRequestHandler_11a6cad5e3b.java)
      at org.apache.tapestry.internal.services.PageRenderDispatcher.process(PageRenderDispatcher.java:97)
      at org.apache.tapestry.internal.services.PageRenderDispatcher.dispatch(PageRenderDispatcher.java:73)
      at $Dispatcher_11a6cad5e42.dispatch($Dispatcher_11a6cad5e42.java)
      at $Dispatcher_11a6cad5e33.dispatch($Dispatcher_11a6cad5e33.java)
      at org.apache.tapestry.services.TapestryModule$13.service(TapestryModule.java:944)
      at de.XYZ.client.services.AppModule$1.service(AppModule.java:92)
      at $RequestFilter_11a6cad5e32.service($RequestFilter_11a6cad5e32.java)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.internal.services.LocalizationFilter.service(LocalizationFilter.java:42)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.services.TapestryModule$3.service(TapestryModule.java:553)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.internal.services.StaticFilesFilter.service(StaticFilesFilter.java:79)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at nu.localhost.tapestry.acegi.services.internal.RequestFilterWrapper$1.doFilter(RequestFilterWrapper.java:60)
      at nu.localhost.tapestry.acegi.services.internal.AcegiExceptionTranslationFilter.doFilter(AcegiExceptionTranslationFilter.java:67)
      at nu.localhost.tapestry.acegi.services.internal.RequestFilterWrapper.service(RequestFilterWrapper.java:54)
      at $RequestFilter_11a6cad5e2e.service($RequestFilter_11a6cad5e2e.java)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.services.TapestryModule$2.service(TapestryModule.java:520)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:93)
      at org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:84)
      at org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:77)
      at org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:106)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at $RequestHandler_11a6cad5e29.service($RequestHandler_11a6cad5e29.java)
      at org.apache.tapestry.services.TapestryModule$12.service(TapestryModule.java:924)
      at de.XYZ.client.services.AppModule$3.service(AppModule.java:152)
      at $HttpServletRequestFilter_11a6cad5e28.service($HttpServletRequestFilter_11a6cad5e28.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at org.apache.tapestry.internal.services.IgnoredPathsFilter.service(IgnoredPathsFilter.java:62)
      at $HttpServletRequestFilter_11a6cad5e27.service($HttpServletRequestFilter_11a6cad5e27.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e26.service($HttpServletRequestFilter_11a6cad5e26.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e25.service($HttpServletRequestFilter_11a6cad5e25.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e24.service($HttpServletRequestFilter_11a6cad5e24.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e23.service($HttpServletRequestFilter_11a6cad5e23.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e22.service($HttpServletRequestFilter_11a6cad5e22.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at $HttpServletRequestHandler_11a6cad5e21.service($HttpServletRequestHandler_11a6cad5e21.java)
      at org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:168)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at com.jamonapi.http.JAMonTomcatValve.invoke(JAMonTomcatValve.java:72)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:354)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:613)

      "catalina-exec-40" Id=96 BLOCKED on java.lang.Class@967f35 owned by "catalina-exec-11" Id=67
      at javassist.ClassPool.toClass2(ClassPool.java:964)
      at javassist.ClassPool.toClass(ClassPool.java:947)
      at org.apache.tapestry.ioc.internal.services.CtClassSource.createClass(CtClassSource.java:87)
      at org.apache.tapestry.ioc.internal.services.AbstractFab.createClass(AbstractFab.java:106)
      at org.apache.tapestry.ioc.internal.ModuleImpl.createProxyInstance(ModuleImpl.java:378)
      at org.apache.tapestry.ioc.internal.ModuleImpl.createProxy(ModuleImpl.java:341)
      at org.apache.tapestry.ioc.internal.ModuleImpl.create(ModuleImpl.java:233)
      at org.apache.tapestry.ioc.internal.ModuleImpl.findOrCreate(ModuleImpl.java:163)
      at org.apache.tapestry.ioc.internal.ModuleImpl.getService(ModuleImpl.java:91)
      at org.apache.tapestry.ioc.internal.RegistryImpl.getService(RegistryImpl.java:293)
      at org.apache.tapestry.ioc.internal.RegistryImpl.getService(RegistryImpl.java:523)
      at org.apache.tapestry.ioc.internal.ObjectLocatorImpl.getService(ObjectLocatorImpl.java:45)
      at org.apache.tapestry.internal.services.ServiceInjectionProvider.provideInjection(ServiceInjectionProvider.java:40)
      at $InjectionProvider_11a6cad5e61.provideInjection($InjectionProvider_11a6cad5e61.java)
      at $InjectionProvider_11a6cad5e59.provideInjection($InjectionProvider_11a6cad5e59.java)
      at org.apache.tapestry.internal.transform.InjectWorker.transform(InjectWorker.java:57)
      at $ComponentClassTransformWorker_11a6cad5e5d.transform($ComponentClassTransformWorker_11a6cad5e5d.java)
      at $ComponentClassTransformWorker_11a6cad5e57.transform($ComponentClassTransformWorker_11a6cad5e57.java)
      at org.apache.tapestry.internal.services.ComponentClassTransformerImpl.transformComponentClass(ComponentClassTransformerImpl.java:147)
      at $ComponentClassTransformer_11a6cad5e1d.transformComponentClass($ComponentClassTransformer_11a6cad5e1d.java)
      at org.apache.tapestry.internal.services.ComponentInstantiatorSourceImpl.onLoad(ComponentInstantiatorSourceImpl.java:166)
      at javassist.Loader.findClass(Loader.java:340)
      at org.apache.tapestry.internal.services.ComponentInstantiatorSourceImpl$PackageAwareLoader.findClass(ComponentInstantiatorSourceImpl.java:75)
      at javassist.Loader.loadClass(Loader.java:311)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
      at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374)
      at de.XYZ.client.components.template.TemplateSessioninfo.getLogoutLink(TemplateSessioninfo.java:25)
      at $PropertyConduit_11a6cad60e2.get($PropertyConduit_11a6cad60e2.java)
      at org.apache.tapestry.internal.bindings.PropBinding.get(PropBinding.java:53)
      at org.apache.tapestry.internal.services.PageElementFactoryImpl$2.provideString(PageElementFactoryImpl.java:136)
      at org.apache.tapestry.internal.services.AttributeExpansionBinding.get(AttributeExpansionBinding.java:37)
      at org.apache.tapestry.internal.structure.InternalComponentResourcesImpl.readParameter(InternalComponentResourcesImpl.java:233)
      at org.apache.tapestry.internal.structure.InternalComponentResourcesImpl.readParameter(InternalComponentResourcesImpl.java:248)
      at org.apache.tapestry.corelib.components.PageLink._$read_parameter_page(PageLink.java)
      at org.apache.tapestry.corelib.components.PageLink.beginRender(PageLink.java:73)
      at org.apache.tapestry.corelib.components.PageLink.beginRender(PageLink.java)
      at org.apache.tapestry.internal.structure.ComponentPageElementImpl$11$1.run(ComponentPageElementImpl.java:338)
      at org.apache.tapestry.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:874)
      at org.apache.tapestry.internal.structure.ComponentPageElementImpl.access$100(ComponentPageElementImpl.java:54)
      at org.apache.tapestry.internal.structure.ComponentPageElementImpl$11.render(ComponentPageElementImpl.java:342)
      at org.apache.tapestry.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:63)
      at org.apache.tapestry.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:84)
      at $PageRenderQueue_11a6cad61e4.render($PageRenderQueue_11a6cad61e4.java)
      at $PageRenderQueue_11a6cad61dd.render($PageRenderQueue_11a6cad61dd.java)
      at org.apache.tapestry.services.TapestryModule$19.renderMarkup(TapestryModule.java:1293)
      at nz.toc.taptoolsapp.services.CustomValidationDecoratorRenderer.renderMarkup(CustomValidationDecoratorRenderer.java:28)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at org.apache.tapestry.services.TapestryModule$23.renderMarkup(TapestryModule.java:1402)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at org.apache.tapestry.services.TapestryModule$22.renderMarkup(TapestryModule.java:1383)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at org.apache.tapestry.services.TapestryModule$21.renderMarkup(TapestryModule.java:1365)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at nz.toc.taptoolsapp.services.AppModule$1.renderMarkup(AppModule.java:79)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at org.apache.tapestry.services.TapestryModule$20.renderMarkup(TapestryModule.java:1347)
      at $MarkupRenderer_11a6cad61e6.renderMarkup($MarkupRenderer_11a6cad61e6.java)
      at $MarkupRenderer_11a6cad61e2.renderMarkup($MarkupRenderer_11a6cad61e2.java)
      at org.apache.tapestry.internal.services.PageMarkupRendererImpl.renderPageMarkup(PageMarkupRendererImpl.java:55)
      at $PageMarkupRenderer_11a6cad61e0.renderPageMarkup($PageMarkupRenderer_11a6cad61e0.java)
      at org.apache.tapestry.internal.services.PageResponseRendererImpl.renderPageResponse(PageResponseRendererImpl.java:57)
      at $PageResponseRenderer_11a6cad5e45.renderPageResponse($PageResponseRenderer_11a6cad5e45.java)
      at org.apache.tapestry.internal.services.PageRenderRequestHandlerImpl.handle(PageRenderRequestHandlerImpl.java:59)
      at org.apache.tapestry.services.TapestryModule$29.handle(TapestryModule.java:1607)
      at $PageRenderRequestHandler_11a6cad5e46.handle($PageRenderRequestHandler_11a6cad5e46.java)
      at $PageRenderRequestHandler_11a6cad5e3b.handle($PageRenderRequestHandler_11a6cad5e3b.java)
      at org.apache.tapestry.internal.services.PageRenderDispatcher.process(PageRenderDispatcher.java:97)
      at org.apache.tapestry.internal.services.PageRenderDispatcher.dispatch(PageRenderDispatcher.java:73)
      at $Dispatcher_11a6cad5e42.dispatch($Dispatcher_11a6cad5e42.java)
      at $Dispatcher_11a6cad5e33.dispatch($Dispatcher_11a6cad5e33.java)
      at org.apache.tapestry.services.TapestryModule$13.service(TapestryModule.java:944)
      at de.XYZ.client.services.AppModule$1.service(AppModule.java:92)
      at $RequestFilter_11a6cad5e32.service($RequestFilter_11a6cad5e32.java)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.internal.services.LocalizationFilter.service(LocalizationFilter.java:42)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.services.TapestryModule$3.service(TapestryModule.java:553)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.internal.services.StaticFilesFilter.service(StaticFilesFilter.java:79)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at nu.localhost.tapestry.acegi.services.internal.RequestFilterWrapper$1.doFilter(RequestFilterWrapper.java:60)
      at nu.localhost.tapestry.acegi.services.internal.AcegiExceptionTranslationFilter.doFilter(AcegiExceptionTranslationFilter.java:67)
      at nu.localhost.tapestry.acegi.services.internal.RequestFilterWrapper.service(RequestFilterWrapper.java:54)
      at $RequestFilter_11a6cad5e2e.service($RequestFilter_11a6cad5e2e.java)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.services.TapestryModule$2.service(TapestryModule.java:520)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:93)
      at org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:84)
      at org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:77)
      at org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:106)
      at $RequestHandler_11a6cad5e34.service($RequestHandler_11a6cad5e34.java)
      at $RequestHandler_11a6cad5e29.service($RequestHandler_11a6cad5e29.java)
      at org.apache.tapestry.services.TapestryModule$12.service(TapestryModule.java:924)
      at de.XYZ.client.services.AppModule$3.service(AppModule.java:152)
      at $HttpServletRequestFilter_11a6cad5e28.service($HttpServletRequestFilter_11a6cad5e28.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at org.apache.tapestry.internal.services.IgnoredPathsFilter.service(IgnoredPathsFilter.java:62)
      at $HttpServletRequestFilter_11a6cad5e27.service($HttpServletRequestFilter_11a6cad5e27.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e26.service($HttpServletRequestFilter_11a6cad5e26.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e25.service($HttpServletRequestFilter_11a6cad5e25.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e24.service($HttpServletRequestFilter_11a6cad5e24.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e23.service($HttpServletRequestFilter_11a6cad5e23.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper$1.doFilter(HttpServletRequestFilterWrapper.java:57)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at nu.localhost.tapestry.acegi.services.internal.HttpServletRequestFilterWrapper.service(HttpServletRequestFilterWrapper.java:52)
      at $HttpServletRequestFilter_11a6cad5e22.service($HttpServletRequestFilter_11a6cad5e22.java)
      at $HttpServletRequestHandler_11a6cad5e2a.service($HttpServletRequestHandler_11a6cad5e2a.java)
      at $HttpServletRequestHandler_11a6cad5e21.service($HttpServletRequestHandler_11a6cad5e21.java)
      at org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:168)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at com.jamonapi.http.JAMonTomcatValve.invoke(JAMonTomcatValve.java:72)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:354)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:613)

      Howard suggested:
      Please add an issue to JIRA. Looks like there's a situation where one thread is transforming classes while another thread is creating PropertyConduit classes. We'll need to create a mutex for this case.

        Issue Links

          Activity

          Hide
          Howard M. Lewis Ship added a comment -

          Looks like a tough one to figure out.

          The first stack trace, for thread catalina-exec-11, is a new class being instantiated inside the component-layer class loader, a PropertyConduit instance (the basis of the prop: binding).

          The second trace, catalina-exec-40 is more odd; looks like it loading a component class and doing some injection along the way; the injection is forcing the creation of a IOC service proxy. That should be locking the web application context class loader (the parent of the component-layer class loader).

          To be honest, I'm not sure I understand what is causing the deadlock; only one thread should be able to load or define a class from a given class loader, but from what I can see, the operation is on two different class loaders.

          I could just make the findClass() method synchronized, but that feels like thrashing, as I don't understand exactly what is causing the problem. I actually don't see how the catlina-exec-11 thread is blocked.

          Still thinking about this, and it still doesn't make sense; we are creating two classes in two different threads, but the new classes are created in two different class loaders (one a parent of the other).

          Show
          Howard M. Lewis Ship added a comment - Looks like a tough one to figure out. The first stack trace, for thread catalina-exec-11, is a new class being instantiated inside the component-layer class loader, a PropertyConduit instance (the basis of the prop: binding). The second trace, catalina-exec-40 is more odd; looks like it loading a component class and doing some injection along the way; the injection is forcing the creation of a IOC service proxy. That should be locking the web application context class loader (the parent of the component-layer class loader). To be honest, I'm not sure I understand what is causing the deadlock; only one thread should be able to load or define a class from a given class loader, but from what I can see, the operation is on two different class loaders. I could just make the findClass() method synchronized, but that feels like thrashing, as I don't understand exactly what is causing the problem. I actually don't see how the catlina-exec-11 thread is blocked. Still thinking about this, and it still doesn't make sense; we are creating two classes in two different threads, but the new classes are created in two different class loaders (one a parent of the other).
          Hide
          Howard M. Lewis Ship added a comment -

          I'm not happy about this fix since I don't actually understand the causes, and don't have any way to reproduce the bug.

          Please retry your tests using the .14 snapshot tommorrow and reopen the bug if you hit another deadlock. I'll be thinking about some other approach that will ensure exclusive access to the class loaders across threads; some shared mutex object. Hopefully it won't come to that, because that's an ugly solution (code that uses ClassFactory would have an additional requirement to manage a mutex).

          Show
          Howard M. Lewis Ship added a comment - I'm not happy about this fix since I don't actually understand the causes, and don't have any way to reproduce the bug. Please retry your tests using the .14 snapshot tommorrow and reopen the bug if you hit another deadlock. I'll be thinking about some other approach that will ensure exclusive access to the class loaders across threads; some shared mutex object. Hopefully it won't come to that, because that's an ugly solution (code that uses ClassFactory would have an additional requirement to manage a mutex).
          Hide
          Joe Hart added a comment -

          Looks like we are running into this deadlock as well in 5.0.11. We tried the patch associated with this issue and still were able to reproduce the deadlock. Any hope of reopening this issue?

          Here is a thread dump showing two deadlocks we have encountered:

          Found one Java-level deadlock:
          =============================
          "http-8080-Processor10":
          waiting to lock monitor 0x08194f04 (object 0x73ba51d8, a org.apache.tapestry.internal.services.ComponentInstantiatorSourceImpl$PackageAwareLoader),
          which is held by "http-8080-Processor9"
          "http-8080-Processor9":
          waiting to lock monitor 0x08192efc (object 0x909ecc08, a java.lang.Class),
          which is held by "http-8080-Processor10"

          And another deadlock we came across:
          Found one Java-level deadlock:
          =============================
          "http-8080-Processor9":
          waiting to lock monitor 0x08a5a838 (object 0x95b84218, a java.lang.Object),
          which is held by "http-8080-Processor8"
          "http-8080-Processor8":
          waiting to lock monitor 0x08a5be7c (object 0x92160ad8, a java.lang.Class),
          which is held by "http-8080-Processor5"
          "http-8080-Processor5":
          waiting to lock monitor 0x8e392558 (object 0x9601bea0, a org.apache.tapestry.internal.services.ComponentInstantiatorSourceImpl$PackageAwareLoader),
          which is held by "http-8080-Processor8"

          Show
          Joe Hart added a comment - Looks like we are running into this deadlock as well in 5.0.11. We tried the patch associated with this issue and still were able to reproduce the deadlock. Any hope of reopening this issue? Here is a thread dump showing two deadlocks we have encountered: Found one Java-level deadlock: ============================= "http-8080-Processor10": waiting to lock monitor 0x08194f04 (object 0x73ba51d8, a org.apache.tapestry.internal.services.ComponentInstantiatorSourceImpl$PackageAwareLoader), which is held by "http-8080-Processor9" "http-8080-Processor9": waiting to lock monitor 0x08192efc (object 0x909ecc08, a java.lang.Class), which is held by "http-8080-Processor10" And another deadlock we came across: Found one Java-level deadlock: ============================= "http-8080-Processor9": waiting to lock monitor 0x08a5a838 (object 0x95b84218, a java.lang.Object), which is held by "http-8080-Processor8" "http-8080-Processor8": waiting to lock monitor 0x08a5be7c (object 0x92160ad8, a java.lang.Class), which is held by "http-8080-Processor5" "http-8080-Processor5": waiting to lock monitor 0x8e392558 (object 0x9601bea0, a org.apache.tapestry.internal.services.ComponentInstantiatorSourceImpl$PackageAwareLoader), which is held by "http-8080-Processor8"
          Hide
          Howard M. Lewis Ship added a comment -

          This fix is in 5.0.14, so there's no reason to reopen it if it is occuring in 5.0.11.

          Show
          Howard M. Lewis Ship added a comment - This fix is in 5.0.14, so there's no reason to reopen it if it is occuring in 5.0.11.
          Hide
          Joe Hart added a comment -

          I looked into your patch (adding the synchronized keyword to ComponentInstantiatorSourceImpl$PackageAwareLoader) and applied that fix into our tapestry implementation. We are still getting the deadlock even with this extra synchronized method so I don't think the problem will be solved in 5.0.14 either. Are there other fixes in 8.0.14 that would affect this deadlock?

          After further testing we are able to reproduce the deadlock with a high consistency by doing the following steps:

          1. Start App
          2. Repeatedly hit a tapestry page with 20-30 concurrent requests. (No other tapestry page can be touched yet...)

          Our pages are quite component heavy (~1500 components) so we are able to click fast enough by hand to reproduce the deadlock. I imagine a lighter page could also trigger the deadlock with enough concurrency produced by a load testing tool.

          Show
          Joe Hart added a comment - I looked into your patch (adding the synchronized keyword to ComponentInstantiatorSourceImpl$PackageAwareLoader) and applied that fix into our tapestry implementation. We are still getting the deadlock even with this extra synchronized method so I don't think the problem will be solved in 5.0.14 either. Are there other fixes in 8.0.14 that would affect this deadlock? After further testing we are able to reproduce the deadlock with a high consistency by doing the following steps: Start App Repeatedly hit a tapestry page with 20-30 concurrent requests. (No other tapestry page can be touched yet...) Our pages are quite component heavy (~1500 components) so we are able to click fast enough by hand to reproduce the deadlock. I imagine a lighter page could also trigger the deadlock with enough concurrency produced by a load testing tool.
          Hide
          Bruno Bonfils added a comment -

          Hi,

          I have the same problem with the deadlock using apache-tomcat (tried both 5 and 6) when I stress our application using jmeter. Most of time the lock occurs with 30 simultaneaous virtual users (with sequential requests, w/o pause), sometimes only with 10 VU :/

          I tried version 14 and 15-SNAPSHOT, got the same problem.

          Thanks

          Show
          Bruno Bonfils added a comment - Hi, I have the same problem with the deadlock using apache-tomcat (tried both 5 and 6) when I stress our application using jmeter. Most of time the lock occurs with 30 simultaneaous virtual users (with sequential requests, w/o pause), sometimes only with 10 VU :/ I tried version 14 and 15-SNAPSHOT, got the same problem. Thanks

            People

            • Assignee:
              Howard M. Lewis Ship
              Reporter:
              jens breitenstein
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development