Jetspeed 2
  1. Jetspeed 2
  2. JS2-689

Spring Bean Factory creation of Prototype (non-singleton) beans causes serious performance degradation under load

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.1.2
    • Component/s: Components Core
    • Labels:
      None

      Description

      It appears that Spring prototype (non-singleton) bean factory creations cause synchronization contention issues under load.
      The NavigationalState and PortalURL beans are created several times per request.
      Under load with JMeter tests, the synchronization of Java Bean support code (in the JDK), called by Spring's bean factory, was causing severe performance degradation.
      Removing this bottleneck improved performance by 5X.
      I've attached the Java source from the package java.beans. I believe its these synchronized methods of the java.beans.PropertyEditorManager class that are causing the contention:

      private static synchronized void initialize() {
      private static synchronized void load(Class targetType, String name) {

      See attached screenshot for performance results before and after removal of prototypes (replaced by constructors).
      Using a 4 processor CPU really brings the contention issue to the forefront.

      When we have hundreds of requests active, there are over 90 blocked threads in the findEditor stack frame:

      Thread t@98: (state = BLOCKED)

      • java.beans.PropertyEditorManager.findEditor(java.lang.Class) @bci=0, line=75 (Compiled frame; information may be imprecise)
      • org.springframework.beans.TypeConverterDelegate.convertIfNecessary(java.lang.String, java.lang.Object, java.lang.Object, java.lang.Class, java.beans.PropertyDescriptor, org.springframework.core.MethodParameter) @bci=115, line=174 (Compiled frame)
      • org.springframework.beans.TypeConverterDelegate.convertIfNecessary(java.lang.Object, java.lang.Class, org.springframework.core.MethodParameter) @bci=7, line=95 (Compiled frame)
      • org.springframework.beans.factory.support.AbstractBeanFactory.doTypeConversionIfNecessary(org.springframework.beans.TypeConverter, java.lang.Object, java.lang.Class, org.springframework.core.MethodParameter) @bci=47, line=761 (Compiled frame)
      • org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(java.lang.String, org.springframework.beans.factory.support.RootBeanDefinition) @bci=191, line=126 (Compiled frame)
      • org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(java.lang.String, org.springframework.beans.factory.support.RootBeanDefinition) @bci=12, line=683 (Compiled frame)
      • org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(java.lang.String, org.springframework.beans.factory.support.RootBeanDefinition, java.lang.Object[]) @bci=224, line=380 (Compiled frame)
      • org.springframework.beans.factory.support.AbstractBeanFactory.getBean(java.lang.String, java.lang.Class, java.lang.Object[]) @bci=362, line=264 (Compiled frame)
      • org.springframework.beans.factory.support.AbstractBeanFactory.getBean(java.lang.String, java.lang.Class) @bci=4, line=160 (Compiled frame)
      • org.apache.jetspeed.container.state.impl.JetspeedNavigationalStateComponent.createURL(javax.servlet.http.HttpServletRequest, java.lang.String) @bci=29, line=131 (Compiled frame)
      • org.apache.jetspeed.pipeline.JetspeedPipeline$Invocation.invokeNex

      My first solution was to remove the Spring prototype code and replace it with straight constructors (replacing commented code below):

      public PortalURL createURL( HttpServletRequest request, String characterEncoding )
      {
      //PortalURL url = (PortalURL) beanFactory.getBean(urlBeanName, PortalURL.class); <--- **** used to be one line prototype creation, commented out....
      NavigationalStateCodec codec = (NavigationalStateCodec)beanFactory.getBean("NavigationalStateCodec");
      JetspeedCache cache = (JetspeedCache)beanFactory.getBean("portletContentCache");
      NavigationalState navState = new SessionFullNavigationalState( codec, cache);
      PortalContext context = (PortalContext)beanFactory.getBean("PortalContext");
      PortalURL url = new PathInfoEncodingPortalURL(navState, context);

      Perhaps it could be configured as:

      public PortalURL createURL( HttpServletRequest request, String characterEncoding )
      {
      if (useSpringToWireBeans)

      { PortalURL url = (PortalURL) beanFactory.getBean(urlBeanName, PortalURL.class); }

      else
      {
      NavigationalStateCodec codec = (NavigationalStateCodec)beanFactory.getBean("NavigationalStateCodec");
      JetspeedCache cache = (JetspeedCache)beanFactory.getBean("portletContentCache");
      NavigationalState navState = new SessionFullNavigationalState( codec, cache);
      PortalContext context = (PortalContext)beanFactory.getBean("PortalContext");
      PortalURL url = new PathInfoEncodingPortalURL(navState, context);

      I will also create an issue in Spring JIRA and see if we can get resolution from the Spring team.....

      1. CPU-Usage-Throughput.tiff
        42 kB
        David Sean Taylor
      2. PropertyEditorManager.java
        6 kB
        David Sean Taylor

        Activity

        Hide
        David Sean Taylor added a comment -

        tested under load. Caching now working correctly.

        Show
        David Sean Taylor added a comment - tested under load. Caching now working correctly.
        Hide
        David Sean Taylor added a comment -

        We have four different Portal URL implementations that can easily be configured with the Spring configuration files.
        If we take this approach above, we would have to require a recompile for everytime someone wanted to switch Portal URL implementations

        Show
        David Sean Taylor added a comment - We have four different Portal URL implementations that can easily be configured with the Spring configuration files. If we take this approach above, we would have to require a recompile for everytime someone wanted to switch Portal URL implementations
        Hide
        Dmitry Bisikalo added a comment -

        David,

        for this particular case you probably don't need to create custom bean factory to handle the creation of Portal URL instances. You could just configure Spring to use a factory method defined someplace (instead of using constructor). This factory method could contain the actual code:

        public PortalURL createPortalURL()

        { NavigationalStateCodec codec = (NavigationalStateCodec)beanFactory.getBean("NavigationalStateCodec"); JetspeedCache cache = (JetspeedCache)beanFactory.getBean("portletContentCache"); NavigationalState navState = new SessionFullNavigationalState( codec, cache); PortalContext context = (PortalContext)beanFactory.getBean("PortalContext"); PortalURL url = new PathInfoEncodingPortalURL(navState, context); }

        This factory method code would encapsulate the optimized logic (i.e. would call "new" directly, as defined above), but from the calling code perspective it would still look like this:

        PortalURL url = (PortalURL) beanFactory.getBean(urlBeanName);

        Having this code encapsulated in the factory method would allow you externalize the logic, but would save you the trouble of creating specialized bean factory.

        Show
        Dmitry Bisikalo added a comment - David, for this particular case you probably don't need to create custom bean factory to handle the creation of Portal URL instances. You could just configure Spring to use a factory method defined someplace (instead of using constructor). This factory method could contain the actual code: public PortalURL createPortalURL() { NavigationalStateCodec codec = (NavigationalStateCodec)beanFactory.getBean("NavigationalStateCodec"); JetspeedCache cache = (JetspeedCache)beanFactory.getBean("portletContentCache"); NavigationalState navState = new SessionFullNavigationalState( codec, cache); PortalContext context = (PortalContext)beanFactory.getBean("PortalContext"); PortalURL url = new PathInfoEncodingPortalURL(navState, context); } This factory method code would encapsulate the optimized logic (i.e. would call "new" directly, as defined above), but from the calling code perspective it would still look like this: PortalURL url = (PortalURL) beanFactory.getBean(urlBeanName); Having this code encapsulated in the factory method would allow you externalize the logic, but would save you the trouble of creating specialized bean factory.
        Hide
        David Sean Taylor added a comment -

        Also see:

        http://opensource.atlassian.com/projects/spring/browse/SPR-3426

        As an immediate patch, we are looking into creating a custom Spring bean factory to handle creation of the Portal URL and NavigationalState prototypes (non-singletons)

        Show
        David Sean Taylor added a comment - Also see: http://opensource.atlassian.com/projects/spring/browse/SPR-3426 As an immediate patch, we are looking into creating a custom Spring bean factory to handle creation of the Portal URL and NavigationalState prototypes (non-singletons)
        Hide
        David Sean Taylor added a comment -

        Screenshot of the graph of CPU Usage and Throughput before and after removing Spring prototypes in request pipeline

        Show
        David Sean Taylor added a comment - Screenshot of the graph of CPU Usage and Throughput before and after removing Spring prototypes in request pipeline
        Hide
        David Sean Taylor added a comment -

        this file is from the Java 1.5 source
        see methods initialize and load

        Show
        David Sean Taylor added a comment - this file is from the Java 1.5 source see methods initialize and load

          People

          • Assignee:
            David Sean Taylor
            Reporter:
            David Sean Taylor
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development