Tapestry
  1. Tapestry
  2. TAPESTRY-1291

Race condition in IoC service creation can create runtime failures

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.0
    • Fix Version/s: 5.0.3
    • Component/s: tapestry-core, tapestry-ioc
    • Labels:
      None

      Description

      Even on my Mac, I occasionally get spurious failures of the integration tests.

      Seems to be some kind of race condition at application startup, where construction of some key service fails, claiming (spuriously) to be dependent on itself.

      The CI server, tapestry.formos.com, seems very succeptible to this.

        Activity

        Howard M. Lewis Ship created issue -
        Hide
        Howard M. Lewis Ship added a comment -

        Seeing this in the logs:

        Construction of service 'tapestry.internal.ActionLinkHandler' has failed due to recursion: the service depends on itself in some way. Please check org.apache.tapestry.internal.services.InternalModule.buildActionLinkHandler(ComponentEventResultProcessor) for references to another service that is itself dependent on service 'tapestry.internal.ActionLinkHandler'.

        With a stack trace of:
        org.apache.tapestry.ioc.internal.OneShotServiceCreator.createObject(OneShotServiceCreator.java:52)
        org.apache.tapestry.internal.services.ComponentActionDispatcher.dispatch(ComponentActionDispatcher.java:115)
        org.apache.tapestry.services.TapestryModule$5.service(TapestryModule.java:479)
        org.apache.tapestry.integration.app1.services.AppModule$1.service(AppModule.java:50)
        org.apache.tapestry.services.TapestryModule$6.service(TapestryModule.java:514)
        org.apache.tapestry.internal.services.StaticFilesFilter.service(StaticFilesFilter.java:56)
        org.apache.tapestry.internal.services.LocalizationFilter.service(LocalizationFilter.java:43)
        org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:91)
        org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:82)
        org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:77)
        org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:104)
        org.apache.tapestry.services.TapestryModule$3.service(TapestryModule.java:402)
        org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:114)
        org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(WebApplicationHandler.java:821)

        (the stack traces now edit out the IoC proxy classes)

        Something is certainly odd here!

        Show
        Howard M. Lewis Ship added a comment - Seeing this in the logs: Construction of service 'tapestry.internal.ActionLinkHandler' has failed due to recursion: the service depends on itself in some way. Please check org.apache.tapestry.internal.services.InternalModule.buildActionLinkHandler(ComponentEventResultProcessor) for references to another service that is itself dependent on service 'tapestry.internal.ActionLinkHandler'. With a stack trace of: org.apache.tapestry.ioc.internal.OneShotServiceCreator.createObject(OneShotServiceCreator.java:52) org.apache.tapestry.internal.services.ComponentActionDispatcher.dispatch(ComponentActionDispatcher.java:115) org.apache.tapestry.services.TapestryModule$5.service(TapestryModule.java:479) org.apache.tapestry.integration.app1.services.AppModule$1.service(AppModule.java:50) org.apache.tapestry.services.TapestryModule$6.service(TapestryModule.java:514) org.apache.tapestry.internal.services.StaticFilesFilter.service(StaticFilesFilter.java:56) org.apache.tapestry.internal.services.LocalizationFilter.service(LocalizationFilter.java:43) org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:91) org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:82) org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:77) org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:104) org.apache.tapestry.services.TapestryModule$3.service(TapestryModule.java:402) org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:114) org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(WebApplicationHandler.java:821) (the stack traces now edit out the IoC proxy classes) Something is certainly odd here!
        Howard M. Lewis Ship made changes -
        Field Original Value New Value
        Assignee Howard M. Lewis Ship [ hlship ]
        Fix Version/s 5.0.3 [ 12312338 ]
        Priority Major [ 3 ] Critical [ 2 ]
        Hide
        Howard M. Lewis Ship added a comment -

        This seems to be gone; I believe it was a combination of factors, including adding some extra synchronization to ClassLoader calls, and giving the integration test suite a chance to "warm up". Also, I defused a kind of race condition by providing an actual favicon.ico.

        Show
        Howard M. Lewis Ship added a comment - This seems to be gone; I believe it was a combination of factors, including adding some extra synchronization to ClassLoader calls, and giving the integration test suite a chance to "warm up". Also, I defused a kind of race condition by providing an actual favicon.ico.
        Howard M. Lewis Ship made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Hide
        Howard M. Lewis Ship added a comment -

        It's back!

        Show
        Howard M. Lewis Ship added a comment - It's back!
        Howard M. Lewis Ship made changes -
        Resolution Fixed [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Howard M. Lewis Ship made changes -
        Summary Flaw in application startup causes continuous integration failures Race condition in IoC service creation can create runtime failures
        Howard M. Lewis Ship made changes -
        Priority Critical [ 2 ] Blocker [ 1 ]
        Hide
        Howard M. Lewis Ship added a comment -

        I have a new working theory: some or all of the errors are derived from a OutOfMemoryError thrown while inside the construction of a service. This is somewhat consistent with what I've been seeing: an error that prevents a service from being instantiated is the only way I can think of that OneShotServiceCreator could get invoked twice (the first call sets the lock and fails, so on a second call, the OSSC is invoked again ... but fails because the lock is set).

        Show
        Howard M. Lewis Ship added a comment - I have a new working theory: some or all of the errors are derived from a OutOfMemoryError thrown while inside the construction of a service. This is somewhat consistent with what I've been seeing: an error that prevents a service from being instantiated is the only way I can think of that OneShotServiceCreator could get invoked twice (the first call sets the lock and fails, so on a second call, the OSSC is invoked again ... but fails because the lock is set).
        Hide
        Howard M. Lewis Ship added a comment -

        Another exception cause just occured to me: I just remembered that CaseInsensitiveMap is decidely not threadsafe (it uses some instance variables for scratch storage).

        Show
        Howard M. Lewis Ship added a comment - Another exception cause just occured to me: I just remembered that CaseInsensitiveMap is decidely not threadsafe (it uses some instance variables for scratch storage).
        Hide
        Howard M. Lewis Ship added a comment -

        Latest batch of changes seem to have been worthwhile., the CI server finally built clean, no problems.

        The OOME exception explains how we could hit a OneShotServiceCreator more than once ... the exception prevented the creation of the service after the lock flag was set, so on retry, we get the "depends on itself" exception.

        Meanwhile, the CastInsensitiveMap was causing much confusion under stress, since it wasn't threadsafe. Under the right circumstances, map.get() != map.get() ... you could ask for an object with key A and get the value for key B.

        I think that's it, and I'll be rolling back a couple of extra syncrhonized blocks before closing the bug. And monitoring the CI server to see if there are any further errors.

        Show
        Howard M. Lewis Ship added a comment - Latest batch of changes seem to have been worthwhile., the CI server finally built clean, no problems. The OOME exception explains how we could hit a OneShotServiceCreator more than once ... the exception prevented the creation of the service after the lock flag was set, so on retry, we get the "depends on itself" exception. Meanwhile, the CastInsensitiveMap was causing much confusion under stress, since it wasn't threadsafe. Under the right circumstances, map.get() != map.get() ... you could ask for an object with key A and get the value for key B. I think that's it, and I'll be rolling back a couple of extra syncrhonized blocks before closing the bug. And monitoring the CI server to see if there are any further errors.
        Hide
        Howard M. Lewis Ship added a comment -

        Well, its survived perhaps a dozen builds without a hitch; I'm satisfied enough to re-close it.

        Show
        Howard M. Lewis Ship added a comment - Well, its survived perhaps a dozen builds without a hitch; I'm satisfied enough to re-close it.
        Howard M. Lewis Ship made changes -
        Resolution Fixed [ 1 ]
        Status Reopened [ 4 ] Closed [ 6 ]
        Mark Thomas made changes -
        Workflow jira [ 12398070 ] Default workflow, editable Closed status [ 12568040 ]
        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12568040 ] jira [ 12591146 ]

          People

          • Assignee:
            Howard M. Lewis Ship
            Reporter:
            Howard M. Lewis Ship
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development