Jetspeed 2
  1. Jetspeed 2
  2. JS2-795

Deadlock while release session on Websphere Release 6.0.2.19

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.1, 2.1.2, 2.1.3, 2.2.0
    • Fix Version/s: 2.1.3, 2.2.0
    • Component/s: Container
    • Labels:
      None

      Description

      We experienced deadlock situation while releasing session on Websphere Release 6.0.2.19 using Jetspeed-2.1. The Problem is also still existent in trunk and occurs in PortalSessionsManagerImpl.

      The attached screenshot documents the deadlock situation.

      Thread Name : Non-deferrable Alarm : 3
      State : Waiting on condition
      Owns Monitor Lock on com.ibm.ws.webcontainer.httpsession.MemorySessionData@53FAA6C8/53FAA6D0
      Java Stack
      at org.apache.jetspeed.container.session.PortletApplicationSessionMonitorImpl.valueUnbound(PortletApplicationSessionMonitorImpl.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionData.processListeners(SessionData.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionData.checkSwappableListeners(SessionData.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionData.sessionUnbound(SessionData.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionContext.notifySessionInvalidated(SessionContext.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionData.invalidate(SessionData.java(Compiled Code))
      ...
      Thread Name : Non-deferrable Alarm : 1
      State : Waiting on condition
      Waiting for Monitor Lock on com.ibm.ws.webcontainer.httpsession.MemorySessionData@53FAA6C8/53FAA6D0
      Owns Monitor Lock on org.apache.jetspeed.container.session.PortalSessionsManagerImpl@31582908/31582910
      Java Stack
      at com.ibm.ws.webcontainer.httpsession.SessionData.invalidate(SessionData.java(Compiled Code))
      at org.apache.jetspeed.container.session.PortletApplicationSessionMonitorImpl.invalidateSession(PortletApplicationSessionMonitorImpl.java(Compiled Code))
      at org.apache.jetspeed.container.session.PortalSessionsManagerImpl.portalSessionDestroyed(PortalSessionsManagerImpl.java(Compiled Code))
      at org.apache.jetspeed.container.session.PortalSessionMonitorImpl.valueUnbound(PortalSessionMonitorImpl.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionData.processListeners(SessionData.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionData.checkSwappableListeners(SessionData.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionData.sessionUnbound(SessionData.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionContext.notifySessionInvalidated(SessionContext.java(Compiled Code))
      at com.ibm.ws.webcontainer.httpsession.SessionData.invalidate(SessionData.java(Compiled Code))
      ...
      ---------------------------

      Other threads pile up because of this situation.
      ---------------------------
      Thread Name : WebContainer : 9
      State : Waiting on monitor
      Waiting for Monitor Lock on org.apache.jetspeed.container.session.PortalSessionsManagerImpl@31582908/31582910
      Java Stack
      at org.apache.jetspeed.container.session.PortalSessionsManagerImpl.checkMonitorSession(PortalSessionsManagerImpl.java(Compiled Code))
      at org.apache.jetspeed.container.JetspeedContainerServlet.doGet(JetspeedContainerServlet.java(Compiled Code))
      at org.apache.jetspeed.container.JetspeedContainerServlet.doPost(JetspeedContainerServlet.java(Compiled Code))
      at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
      at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))

      What happens:

      1.) thread1: Websphere releases the session, LOCKS the SessionData and PortletApplicationSessionMonitorImpl.valueUnbound is called

      2.) thread2: <synchronized> PortalSessionsManagerImpl.portalSessionDestroyed is called by PortalSessionMonitorImpl.valueUnbound (called by the Container as well) and waits for lock on SessionData

      3.) thread1: PortletApplicationSessionMonitorImpl.valueUnbound tries to call <synchronized> PortletApplicationSessionMonitorImpl.sessionDestroyed which is blocked by thread2

      The patch supplied removes all synchronized methods from the PortalSessionsManagerImpl. This can be done because the Maps sessionMonitors and portalSessionsRegistry are SynchronizedMaps. The iteration over the elements of the maps is done via a shallow copy of the SynchronizedMaps, so a change of the original Map does not affect the iteration process.

      1. PortalSessionsManagerImpl.java.diff
        7 kB
        Joachim Müller
      2. debug-information.gif
        97 kB
        Joachim Müller

        Activity

        Hide
        Woonsan Ko added a comment -

        Applied Joachim Müller's patch.
        I've done some load testings under Tomcat and WebSphere 6.1.0.11 and confirmed no side effect.

        Cheers!

        Show
        Woonsan Ko added a comment - Applied Joachim Müller's patch. I've done some load testings under Tomcat and WebSphere 6.1.0.11 and confirmed no side effect. Cheers!
        Hide
        Joachim Müller added a comment -

        Exactly that's what I think. The situation happend under very heavy load.

        Show
        Joachim Müller added a comment - Exactly that's what I think. The situation happend under very heavy load.
        Hide
        Woonsan Ko added a comment -

        I agree with you, Joachim. I understand how SynchronizedMap and SynchronizedCollection work. Thank you very much!

        By the way, I guess that your deadlock situation happened when a portal session and portlet session(s) are expired almost at the same time by the container. Is my guess right?

        Show
        Woonsan Ko added a comment - I agree with you, Joachim. I understand how SynchronizedMap and SynchronizedCollection work. Thank you very much! By the way, I guess that your deadlock situation happened when a portal session and portlet session(s) are expired almost at the same time by the container. Is my guess right?
        Hide
        Joachim Müller added a comment -

        Thanks for checking the patch! Since this patch is at the very base of Jetspeed we all should look twice at it.

        In this case I think we do not need extra synchronization. If you have a look at the source of the SynchronizedMap inside Collections the methods SynchronizedMap.values is synchronized against an internal monitor (mutex). SynchronizedMap.values results in a SynchronizedCollection and the SynchronizedCollection.toArray() is again synchronized against the same mutex. So all methods to create a shallow copy are synchronized. No need to synchronized it again.

        Show
        Joachim Müller added a comment - Thanks for checking the patch! Since this patch is at the very base of Jetspeed we all should look twice at it. In this case I think we do not need extra synchronization. If you have a look at the source of the SynchronizedMap inside Collections the methods SynchronizedMap.values is synchronized against an internal monitor (mutex). SynchronizedMap.values results in a SynchronizedCollection and the SynchronizedCollection.toArray() is again synchronized against the same mutex. So all methods to create a shallow copy are synchronized. No need to synchronized it again.
        Hide
        Woonsan Ko added a comment -

        Thanks for the patch, Joachim.
        IMHO, there remains some synchronization issues in your patch.

        According to javadoc for java.util.Collections#synchronizedMap(), it is imperative that the user manually synchronize on the returned map when iterating over any of its collection views.
        So, I think the followings could be problematic:

        + Iterator iter = valuesShallowCopy(psr.sessionMonitors.values()).iterator();
        <snip>
        + private Collection valuesShallowCopy(Collection inValues)

        { + return Arrays.asList(inValues.toArray()); + }

        The values() method involves iterator() call. Also, inValues.toArray() involves iterations.
        Therefore, I think it can throw ConcurrentModificationExceptions.

        Show
        Woonsan Ko added a comment - Thanks for the patch, Joachim. IMHO, there remains some synchronization issues in your patch. According to javadoc for java.util.Collections#synchronizedMap(), it is imperative that the user manually synchronize on the returned map when iterating over any of its collection views. So, I think the followings could be problematic: + Iterator iter = valuesShallowCopy(psr.sessionMonitors.values()).iterator(); <snip> + private Collection valuesShallowCopy(Collection inValues) { + return Arrays.asList(inValues.toArray()); + } The values() method involves iterator() call. Also, inValues.toArray() involves iterations. Therefore, I think it can throw ConcurrentModificationExceptions.
        Hide
        Joachim Müller added a comment -

        TYPO:

        3.) thread1: PortletApplicationSessionMonitorImpl.valueUnbound tries to call <synchronized> PortletApplicationSessionMonitorImpl.sessionDestroyed which is blocked by thread2

        should be

        3.) thread1: PortletApplicationSessionMonitorImpl.valueUnbound tries to call <synchronized> PortalSessionsManagerImpl.sessionDestroyed which is blocked by thread2

        Show
        Joachim Müller added a comment - TYPO: 3.) thread1: PortletApplicationSessionMonitorImpl.valueUnbound tries to call <synchronized> PortletApplicationSessionMonitorImpl.sessionDestroyed which is blocked by thread2 should be 3.) thread1: PortletApplicationSessionMonitorImpl.valueUnbound tries to call <synchronized> PortalSessionsManagerImpl.sessionDestroyed which is blocked by thread2
        Hide
        Joachim Müller added a comment -

        the patch

        Show
        Joachim Müller added a comment - the patch
        Hide
        Joachim Müller added a comment -

        screenshot with debug information

        Show
        Joachim Müller added a comment - screenshot with debug information

          People

          • Assignee:
            Woonsan Ko
            Reporter:
            Joachim Müller
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development