Uploaded image for project: 'Tapestry'
  1. Tapestry
  2. TAPESTRY-1291

Race condition in IoC service creation can create runtime failures

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.0
    • Fix Version/s: 5.0.3
    • Component/s: tapestry-core, tapestry-ioc
    • Labels:
      None

      Description

      Even on my Mac, I occasionally get spurious failures of the integration tests.

      Seems to be some kind of race condition at application startup, where construction of some key service fails, claiming (spuriously) to be dependent on itself.

      The CI server, tapestry.formos.com, seems very succeptible to this.

        Activity

        Hide
        hlship Howard M. Lewis Ship added a comment -

        Well, its survived perhaps a dozen builds without a hitch; I'm satisfied enough to re-close it.

        Show
        hlship Howard M. Lewis Ship added a comment - Well, its survived perhaps a dozen builds without a hitch; I'm satisfied enough to re-close it.
        Hide
        hlship Howard M. Lewis Ship added a comment -

        Latest batch of changes seem to have been worthwhile., the CI server finally built clean, no problems.

        The OOME exception explains how we could hit a OneShotServiceCreator more than once ... the exception prevented the creation of the service after the lock flag was set, so on retry, we get the "depends on itself" exception.

        Meanwhile, the CastInsensitiveMap was causing much confusion under stress, since it wasn't threadsafe. Under the right circumstances, map.get() != map.get() ... you could ask for an object with key A and get the value for key B.

        I think that's it, and I'll be rolling back a couple of extra syncrhonized blocks before closing the bug. And monitoring the CI server to see if there are any further errors.

        Show
        hlship Howard M. Lewis Ship added a comment - Latest batch of changes seem to have been worthwhile., the CI server finally built clean, no problems. The OOME exception explains how we could hit a OneShotServiceCreator more than once ... the exception prevented the creation of the service after the lock flag was set, so on retry, we get the "depends on itself" exception. Meanwhile, the CastInsensitiveMap was causing much confusion under stress, since it wasn't threadsafe. Under the right circumstances, map.get() != map.get() ... you could ask for an object with key A and get the value for key B. I think that's it, and I'll be rolling back a couple of extra syncrhonized blocks before closing the bug. And monitoring the CI server to see if there are any further errors.
        Hide
        hlship Howard M. Lewis Ship added a comment -

        Another exception cause just occured to me: I just remembered that CaseInsensitiveMap is decidely not threadsafe (it uses some instance variables for scratch storage).

        Show
        hlship Howard M. Lewis Ship added a comment - Another exception cause just occured to me: I just remembered that CaseInsensitiveMap is decidely not threadsafe (it uses some instance variables for scratch storage).
        Hide
        hlship Howard M. Lewis Ship added a comment -

        I have a new working theory: some or all of the errors are derived from a OutOfMemoryError thrown while inside the construction of a service. This is somewhat consistent with what I've been seeing: an error that prevents a service from being instantiated is the only way I can think of that OneShotServiceCreator could get invoked twice (the first call sets the lock and fails, so on a second call, the OSSC is invoked again ... but fails because the lock is set).

        Show
        hlship Howard M. Lewis Ship added a comment - I have a new working theory: some or all of the errors are derived from a OutOfMemoryError thrown while inside the construction of a service. This is somewhat consistent with what I've been seeing: an error that prevents a service from being instantiated is the only way I can think of that OneShotServiceCreator could get invoked twice (the first call sets the lock and fails, so on a second call, the OSSC is invoked again ... but fails because the lock is set).
        Hide
        hlship Howard M. Lewis Ship added a comment -

        It's back!

        Show
        hlship Howard M. Lewis Ship added a comment - It's back!
        Hide
        hlship Howard M. Lewis Ship added a comment -

        This seems to be gone; I believe it was a combination of factors, including adding some extra synchronization to ClassLoader calls, and giving the integration test suite a chance to "warm up". Also, I defused a kind of race condition by providing an actual favicon.ico.

        Show
        hlship Howard M. Lewis Ship added a comment - This seems to be gone; I believe it was a combination of factors, including adding some extra synchronization to ClassLoader calls, and giving the integration test suite a chance to "warm up". Also, I defused a kind of race condition by providing an actual favicon.ico.
        Hide
        hlship Howard M. Lewis Ship added a comment -

        Seeing this in the logs:

        Construction of service 'tapestry.internal.ActionLinkHandler' has failed due to recursion: the service depends on itself in some way. Please check org.apache.tapestry.internal.services.InternalModule.buildActionLinkHandler(ComponentEventResultProcessor) for references to another service that is itself dependent on service 'tapestry.internal.ActionLinkHandler'.

        With a stack trace of:
        org.apache.tapestry.ioc.internal.OneShotServiceCreator.createObject(OneShotServiceCreator.java:52)
        org.apache.tapestry.internal.services.ComponentActionDispatcher.dispatch(ComponentActionDispatcher.java:115)
        org.apache.tapestry.services.TapestryModule$5.service(TapestryModule.java:479)
        org.apache.tapestry.integration.app1.services.AppModule$1.service(AppModule.java:50)
        org.apache.tapestry.services.TapestryModule$6.service(TapestryModule.java:514)
        org.apache.tapestry.internal.services.StaticFilesFilter.service(StaticFilesFilter.java:56)
        org.apache.tapestry.internal.services.LocalizationFilter.service(LocalizationFilter.java:43)
        org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:91)
        org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:82)
        org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:77)
        org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:104)
        org.apache.tapestry.services.TapestryModule$3.service(TapestryModule.java:402)
        org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:114)
        org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(WebApplicationHandler.java:821)

        (the stack traces now edit out the IoC proxy classes)

        Something is certainly odd here!

        Show
        hlship Howard M. Lewis Ship added a comment - Seeing this in the logs: Construction of service 'tapestry.internal.ActionLinkHandler' has failed due to recursion: the service depends on itself in some way. Please check org.apache.tapestry.internal.services.InternalModule.buildActionLinkHandler(ComponentEventResultProcessor) for references to another service that is itself dependent on service 'tapestry.internal.ActionLinkHandler'. With a stack trace of: org.apache.tapestry.ioc.internal.OneShotServiceCreator.createObject(OneShotServiceCreator.java:52) org.apache.tapestry.internal.services.ComponentActionDispatcher.dispatch(ComponentActionDispatcher.java:115) org.apache.tapestry.services.TapestryModule$5.service(TapestryModule.java:479) org.apache.tapestry.integration.app1.services.AppModule$1.service(AppModule.java:50) org.apache.tapestry.services.TapestryModule$6.service(TapestryModule.java:514) org.apache.tapestry.internal.services.StaticFilesFilter.service(StaticFilesFilter.java:56) org.apache.tapestry.internal.services.LocalizationFilter.service(LocalizationFilter.java:43) org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:91) org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:82) org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:77) org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:104) org.apache.tapestry.services.TapestryModule$3.service(TapestryModule.java:402) org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:114) org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(WebApplicationHandler.java:821) (the stack traces now edit out the IoC proxy classes) Something is certainly odd here!

          People

          • Assignee:
            hlship Howard M. Lewis Ship
            Reporter:
            hlship Howard M. Lewis Ship
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development