Jetspeed 2
  1. Jetspeed 2
  2. JS2-613

ConcurrentModificationException under high load

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1-dev
    • Fix Version/s: 2.1
    • Component/s: PSML
    • Labels:
      None
    • Environment:
      Java 1.5 , Oracle 9

      Description

      Get this exception under high load. Seems to be a concurrency issue around the nodes List passed to the constructor of NodeSetImpl. Help on this would be greatly appreciated.

      2006-11-15 06:48:44,591 [TP-Processor8] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/portal].[jetspeed] - Servlet.service() for servlet jetspeed threw exception
      org.apache.jetspeed.pipeline.PipelineException: org.apache.jetspeed.pipeline.PipelineException: org.apache.jetspeed.pipeline.PipelineException: java.util.ConcurrentModificationException
      at org.apache.jetspeed.security.impl.PasswordCredentialValveImpl.invoke(PasswordCredentialValveImpl.java:154)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at com.covisint.cep.jetspeed.pipeline.LocalizationValveImpl.invoke(LocalizationValveImpl.java:75)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at org.apache.jetspeed.security.impl.AbstractSecurityValve$1.run(AbstractSecurityValve.java:117)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAsPrivileged(Subject.java:454)
      at org.apache.jetspeed.security.impl.AbstractSecurityValve.invoke(AbstractSecurityValve.java:111)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at org.apache.jetspeed.container.url.impl.PortalURLValveImpl.invoke(PortalURLValveImpl.java:67)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at com.covisint.cep.jetspeed.pipeline.NetworkPortValveImpl.invoke(NetworkPortValveImpl.java:50)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at com.covisint.cep.jetspeed.pipeline.LastUrlCookieValveImpl.invoke(LastUrlCookieValveImpl.java:58)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at com.covisint.cep.jetspeed.pipeline.ContentPreviewValveImpl.invoke(ContentPreviewValveImpl.java:48)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at org.apache.jetspeed.capabilities.impl.CapabilityValveImpl.invoke(CapabilityValveImpl.java:128)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at org.apache.jetspeed.pipeline.JetspeedPipeline.invoke(JetspeedPipeline.java:145)
      at org.apache.jetspeed.engine.JetspeedEngine.service(JetspeedEngine.java:214)
      at org.apache.jetspeed.engine.JetspeedServlet.doGet(JetspeedServlet.java:241)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
      at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
      at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
      at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
      at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: org.apache.jetspeed.pipeline.PipelineException: org.apache.jetspeed.pipeline.PipelineException: java.util.ConcurrentModificationException
      at org.apache.jetspeed.security.impl.LoginValidationValveImpl.invoke(LoginValidationValveImpl.java:164)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at org.apache.jetspeed.security.impl.PasswordCredentialValveImpl.invoke(PasswordCredentialValveImpl.java:149)
      ... 39 more
      Caused by: org.apache.jetspeed.pipeline.PipelineException: java.util.ConcurrentModificationException
      at org.apache.jetspeed.profiler.impl.ProfilerValveImpl.invoke(ProfilerValveImpl.java:304)
      at org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNext(JetspeedPipeline.java:166)
      at org.apache.jetspeed.security.impl.LoginValidationValveImpl.invoke(LoginValidationValveImpl.java:159)
      ... 41 more
      Caused by: java.util.ConcurrentModificationException
      at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:449)
      at java.util.AbstractList$Itr.next(AbstractList.java:420)
      at org.apache.jetspeed.page.document.impl.NodeSetImpl.<init>(NodeSetImpl.java:50)
      at org.apache.jetspeed.om.folder.impl.FolderImpl.getAllNodeSet(FolderImpl.java:1024)
      at org.apache.jetspeed.om.folder.impl.FolderImpl.getAll(FolderImpl.java:838)
      at org.apache.jetspeed.page.impl.DatabasePageManager.getAll(DatabasePageManager.java:925)
      at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:284)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:155)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:122)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:56)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
      at org.apache.jetspeed.util.interceptors.PageManagerInterceptor.invoke(PageManagerInterceptor.java:46)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174)
      at $Proxy4.getAll(Unknown Source)
      at org.apache.jetspeed.om.folder.impl.FolderImpl.getAll(FolderImpl.java:833)
      at org.apache.jetspeed.om.folder.proxy.FolderProxy.aggregateChildren(FolderProxy.java:763)
      at org.apache.jetspeed.om.folder.proxy.FolderProxy.getAll(FolderProxy.java:309)
      at org.apache.jetspeed.om.folder.proxy.FolderProxy.getPages(FolderProxy.java:450)
      at org.apache.jetspeed.om.folder.proxy.FolderProxy.getPage(FolderProxy.java:473)
      at org.apache.jetspeed.om.folder.proxy.FolderProxy.selectDefaultPageFromAggregateFolders(FolderProxy.java:715)
      at org.apache.jetspeed.om.folder.proxy.FolderProxy.getDefaultPage(FolderProxy.java:324)
      at org.apache.jetspeed.om.folder.proxy.FolderProxy.invoke(FolderProxy.java:203)
      at $Proxy14.getDefaultPage(Unknown Source)
      at org.apache.jetspeed.portalsite.impl.PortalSiteSessionContextImpl.selectRequestPage(PortalSiteSessionContextImpl.java:465)
      at org.apache.jetspeed.portalsite.impl.PortalSiteSessionContextImpl.selectRequestPage(PortalSiteSessionContextImpl.java:221)
      at org.apache.jetspeed.portalsite.impl.PortalSiteRequestContextImpl.getPage(PortalSiteRequestContextImpl.java:212)
      at org.apache.jetspeed.portalsite.impl.PortalSiteRequestContextImpl.getManagedPage(PortalSiteRequestContextImpl.java:195)
      at org.apache.jetspeed.profiler.impl.ProfilerValveImpl.invoke(ProfilerValveImpl.java:248)
      ... 43 more

        Activity

        Hide
        David Sean Taylor added a comment -

        patch applied

        Show
        David Sean Taylor added a comment - patch applied
        Hide
        Ethan Adams added a comment -

        Unfortunately, the NodeSetImpl fix didn't quite get it. Further testing showed an ArrayIndexOutOfBoundsException from under extreme load.

        So, I've added synchronization around List all.

        public List accessAll()
        {
        // create initial collection if necessary
        if (all == null)

        { all = java.util.Collections.synchronizedList(DatabasePageManagerUtils.createList()); }

        return all;
        }

        In resetAll(boolean)

        synchronized(all)
        {
        Iterator nodeIter = accessAll().iterator();
        while (nodeIter.hasNext())
        {
        Node node = (Node)nodeIter.next();
        if (node instanceof PageImpl)

        { pages.add(node); }

        else if (node instanceof FolderImpl)

        { folders.add(node); }

        else if (node instanceof LinkImpl)

        { links.add(node); }

        else if (node instanceof PageSecurityImpl)

        { pageSecurity = (PageSecurityImpl)node; }

        }
        }

        private NodeSet getAllNodeSet()
        {
        if (allNodeSet == null)
        {
        if ((all != null) && !all.isEmpty())
        {
        List allCopy = new java.util.ArrayList();
        synchronized(all)

        { allCopy.addAll(all); }

        allNodeSet = new NodeSetImpl(allCopy, createDocumentOrderComparator());
        }
        else

        { allNodeSet = NodeSetImpl.EMPTY_NODE_SET; }

        }
        return allNodeSet;
        }

        I don't believe the NodeSetImpl change is necessary with the above change, but it couldn't hurt either.

        Show
        Ethan Adams added a comment - Unfortunately, the NodeSetImpl fix didn't quite get it. Further testing showed an ArrayIndexOutOfBoundsException from under extreme load. So, I've added synchronization around List all. public List accessAll() { // create initial collection if necessary if (all == null) { all = java.util.Collections.synchronizedList(DatabasePageManagerUtils.createList()); } return all; } In resetAll(boolean) synchronized(all) { Iterator nodeIter = accessAll().iterator(); while (nodeIter.hasNext()) { Node node = (Node)nodeIter.next(); if (node instanceof PageImpl) { pages.add(node); } else if (node instanceof FolderImpl) { folders.add(node); } else if (node instanceof LinkImpl) { links.add(node); } else if (node instanceof PageSecurityImpl) { pageSecurity = (PageSecurityImpl)node; } } } private NodeSet getAllNodeSet() { if (allNodeSet == null) { if ((all != null) && !all.isEmpty()) { List allCopy = new java.util.ArrayList(); synchronized(all) { allCopy.addAll(all); } allNodeSet = new NodeSetImpl(allCopy, createDocumentOrderComparator()); } else { allNodeSet = NodeSetImpl.EMPTY_NODE_SET; } } return allNodeSet; } I don't believe the NodeSetImpl change is necessary with the above change, but it couldn't hurt either.
        Hide
        David Sean Taylor added a comment -

        reopening as more concurrency issues found

        Show
        David Sean Taylor added a comment - reopening as more concurrency issues found
        Hide
        David Sean Taylor added a comment -

        Could you please send a proper patch in the future?
        I can't even find the original patch I applied, its not here, so it must have come over the dev list...
        2. Which file do I apply this patch to?
        best i can tell,its FolderImpl.java (db-psml)
        I'll start applying it there....

        Show
        David Sean Taylor added a comment - Could you please send a proper patch in the future? I can't even find the original patch I applied, its not here, so it must have come over the dev list... 2. Which file do I apply this patch to? best i can tell,its FolderImpl.java (db-psml) I'll start applying it there....
        Hide
        David Sean Taylor added a comment -

        patch applied, please retest

        Show
        David Sean Taylor added a comment - patch applied, please retest
        Hide
        Ethan Adams added a comment -

        fixes resolved problem

        Show
        Ethan Adams added a comment - fixes resolved problem
        Hide
        David Sean Taylor added a comment -

        Im going to resolve this issue for 2.1, as we are out of time.
        I made another synchronization pass over the components
        Ethan, if you have anything else that I missed, please let me know

        Show
        David Sean Taylor added a comment - Im going to resolve this issue for 2.1, as we are out of time. I made another synchronization pass over the components Ethan, if you have anything else that I missed, please let me know

          People

          • Assignee:
            David Sean Taylor
            Reporter:
            Ethan Adams
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development