Uploaded image for project: 'Karaf'
  1. Karaf
  2. KARAF-1715

Occasional freeze on feature:install webconsole

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0.RC1, 3.0.0
    • Fix Version/s: 2.2.10, 2.3.1, 3.0.0
    • Component/s: karaf-webconsole
    • Labels:
      None

      Description

      This is not something that happens all the time, so it might require a couple of attempts in order to reproduce it.

      On a clean Karaf container feature:install webconsole can freeze and never return. When this happens CTRL-C does not stop the command execution and the only way to recover is a restart of the container.

      1. karaf.log
        47 kB
        Guillaume Nodet

        Issue Links

          Activity

          Hide
          achim_nierbeck Achim Nierbeck added a comment -

          hey Ioannis, you think it's possible to try to create a paxexam-karaf itest for this behaviour?
          Maybe it's a timing issues of the features ....

          Show
          achim_nierbeck Achim Nierbeck added a comment - hey Ioannis, you think it's possible to try to create a paxexam-karaf itest for this behaviour? Maybe it's a timing issues of the features ....
          Hide
          chris@die-schneider.net Christian Schneider added a comment -

          I found that a thread named CM Configuration Updater (ManagedService Update: pid=org.ops4j.pax.web)
          hangs on object.wait
          Daemon Thread [CM Configuration Updater (ManagedService Update: pid=org.ops4j.pax.web)] (Suspended)
          Object.wait(long) line: not available [native method]
          Object[](Object).wait() line: 485 [local variables unavailable]
          Felix.acquireBundleLock(BundleImpl, int) line: 4871
          Felix.registerService(BundleImpl, String[], Object, Dictionary) line: 3205
          BundleContextImpl.registerService(String[], Object, Dictionary) line: 346
          BundleContextImpl.registerService(String, Object, Dictionary) line: 320
          JettyServerWrapper.addContext(Model) line: 225
          JettyServerWrapper.getOrCreateContext(Model) line: 114
          JettyServerImpl.addServlet(ServletModel) line: 163
          ServerControllerImpl$Started.addServlet(ServletModel) line: 277
          ServerControllerImpl.addServlet(ServletModel) line: 112
          HttpServiceStarted$1.stateChanged(ServerEvent) line: 123
          ServerControllerImpl.notifyListeners(ServerEvent) line: 206
          ServerControllerImpl$Stopped.start() line: 522
          ServerControllerImpl.start() line: 72
          ServerControllerImpl$Started.configure() line: 272
          ServerControllerImpl.configure(Configuration) line: 89
          Activator$3.updated(Dictionary) line: 236
          ConfigurationManager$ManagedServiceUpdate.run() line: 1467
          UpdateThread.run() line: 103
          Thread.run() line: 662

          Show
          chris@die-schneider.net Christian Schneider added a comment - I found that a thread named CM Configuration Updater (ManagedService Update: pid=org.ops4j.pax.web) hangs on object.wait Daemon Thread [CM Configuration Updater (ManagedService Update: pid=org.ops4j.pax.web)] (Suspended) Object.wait(long) line: not available [native method] Object[](Object).wait() line: 485 [local variables unavailable] Felix.acquireBundleLock(BundleImpl, int) line: 4871 Felix.registerService(BundleImpl, String[], Object, Dictionary) line: 3205 BundleContextImpl.registerService(String[], Object, Dictionary) line: 346 BundleContextImpl.registerService(String, Object, Dictionary) line: 320 JettyServerWrapper.addContext(Model) line: 225 JettyServerWrapper.getOrCreateContext(Model) line: 114 JettyServerImpl.addServlet(ServletModel) line: 163 ServerControllerImpl$Started.addServlet(ServletModel) line: 277 ServerControllerImpl.addServlet(ServletModel) line: 112 HttpServiceStarted$1.stateChanged(ServerEvent) line: 123 ServerControllerImpl.notifyListeners(ServerEvent) line: 206 ServerControllerImpl$Stopped.start() line: 522 ServerControllerImpl.start() line: 72 ServerControllerImpl$Started.configure() line: 272 ServerControllerImpl.configure(Configuration) line: 89 Activator$3.updated(Dictionary) line: 236 ConfigurationManager$ManagedServiceUpdate.run() line: 1467 UpdateThread.run() line: 103 Thread.run() line: 662
          Hide
          chris@die-schneider.net Christian Schneider added a comment -

          Here is another stack trace of a thread that may be involved.
          So it seems because of the waitForInit the activator thread does not return. As the activator probably already locked the bundle the config admin thread above will not succeed. So the waitForIntit can also not succeed.

          Daemon Thread [Thread-8] (Suspended)
          Thread.sleep(long) line: not available [native method]
          HttpServiceStarted.waitForInit(HttpServiceStarted$ServletPlus) line: 268
          HttpServiceStarted.registerServlet(String, Servlet, Dictionary, HttpContext) line: 193
          HttpServiceProxy.registerServlet(String, Servlet, Dictionary, HttpContext) line: 59
          KarafOsgiManager(OsgiManager).bindHttpService(HttpService) line: 726
          OsgiManager$HttpServiceTracker.addingService(ServiceReference) line: 650
          ServiceTracker$Tracked.customizerAdding(ServiceReference, ServiceEvent) line: 980
          ServiceTracker$Tracked.customizerAdding(Object, Object) line: 906
          ServiceTracker$Tracked(AbstractTracked).trackAdding(Object, Object) line: 262
          ServiceTracker$Tracked(AbstractTracked).trackInitial() line: 185
          OsgiManager$HttpServiceTracker(ServiceTracker).open(boolean) line: 348
          OsgiManager$HttpServiceTracker(ServiceTracker).open() line: 283
          KarafOsgiManager(OsgiManager).updateConfiguration(Dictionary) line: 869
          KarafOsgiManager(OsgiManager).<init>(BundleContext) line: 287
          KarafOsgiManager.<init>(BundleContext) line: 37
          KarafOsgiManagerActivator$1.run() line: 45
          KarafOsgiManagerActivator.start(BundleContext) line: 49
          SecureAction.startActivator(BundleActivator, BundleContext) line: 645
          Felix.activateBundle(BundleImpl, boolean) line: 1977
          Felix.startBundle(BundleImpl, int) line: 1895
          BundleImpl.start(int) line: 944
          BundleImpl.start() line: 931
          FeaturesServiceImpl.startBundle(FeaturesServiceImpl$InstallationState, Bundle) line: 540
          FeaturesServiceImpl.installFeatures(Set<Feature>, EnumSet<Option>) line: 495
          FeaturesServiceImpl.installFeature(Feature, EnumSet<Option>) line: 431
          FeaturesServiceImpl.installFeature(String, String, EnumSet<Option>) line: 420
          Proxyb7a59f09_5bd1_4db1_9436_b2f6b4e1402b.installFeature(String, String, EnumSet) line: not available
          InstallFeatureCommand.doExecute(FeaturesService) line: 62
          InstallFeatureCommand(FeaturesCommandSupport).doExecute() line: 38
          InstallFeatureCommand(AbstractAction).execute(CommandSession) line: 33
          InstallFeatureCommand(OsgiCommandSupport).execute(CommandSession) line: 39
          BlueprintCommand(AbstractCommand).execute(CommandSession, List<Object>) line: 33
          NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
          NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
          DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
          Method.invoke(Object, Object...) line: 597
          ProxyHandler$1.invoke(Object, Method, Object[]) line: 54
          ProxyHandler.invoke(Object, Method, Object[]) line: 119
          $BlueprintCommand25912032.execute(CommandSession, List<Object>) line: not available
          CommandProxy.execute(CommandSession, List<Object>) line: 78
          Closure.executeCmd(String, List<Object>) line: 477
          Closure.executeStatement(List<Token>) line: 403
          Pipe.run() line: 108
          Closure.execute(List<Object>) line: 183
          Closure.execute(CommandSession, List<Object>) line: 120
          CommandSessionImpl.execute(CharSequence) line: 89
          ConsoleImpl.run() line: 179
          ConsoleFactoryService$3(Thread).run() line: 662
          ConsoleFactoryService$3.doRun() line: 115
          ConsoleFactoryService$3$1.run() line: 106
          AccessController.doPrivileged(PrivilegedAction<T>, AccessControlContext) line: not available [native method]
          Subject.doAs(Subject, PrivilegedAction<T>) line: 337
          ConsoleFactoryService$3.run() line: 104

          Show
          chris@die-schneider.net Christian Schneider added a comment - Here is another stack trace of a thread that may be involved. So it seems because of the waitForInit the activator thread does not return. As the activator probably already locked the bundle the config admin thread above will not succeed. So the waitForIntit can also not succeed. Daemon Thread [Thread-8] (Suspended) Thread.sleep(long) line: not available [native method] HttpServiceStarted.waitForInit(HttpServiceStarted$ServletPlus) line: 268 HttpServiceStarted.registerServlet(String, Servlet, Dictionary, HttpContext) line: 193 HttpServiceProxy.registerServlet(String, Servlet, Dictionary, HttpContext) line: 59 KarafOsgiManager(OsgiManager).bindHttpService(HttpService) line: 726 OsgiManager$HttpServiceTracker.addingService(ServiceReference) line: 650 ServiceTracker$Tracked.customizerAdding(ServiceReference, ServiceEvent) line: 980 ServiceTracker$Tracked.customizerAdding(Object, Object) line: 906 ServiceTracker$Tracked(AbstractTracked).trackAdding(Object, Object) line: 262 ServiceTracker$Tracked(AbstractTracked).trackInitial() line: 185 OsgiManager$HttpServiceTracker(ServiceTracker).open(boolean) line: 348 OsgiManager$HttpServiceTracker(ServiceTracker).open() line: 283 KarafOsgiManager(OsgiManager).updateConfiguration(Dictionary) line: 869 KarafOsgiManager(OsgiManager).<init>(BundleContext) line: 287 KarafOsgiManager.<init>(BundleContext) line: 37 KarafOsgiManagerActivator$1.run() line: 45 KarafOsgiManagerActivator.start(BundleContext) line: 49 SecureAction.startActivator(BundleActivator, BundleContext) line: 645 Felix.activateBundle(BundleImpl, boolean) line: 1977 Felix.startBundle(BundleImpl, int) line: 1895 BundleImpl.start(int) line: 944 BundleImpl.start() line: 931 FeaturesServiceImpl.startBundle(FeaturesServiceImpl$InstallationState, Bundle) line: 540 FeaturesServiceImpl.installFeatures(Set<Feature>, EnumSet<Option>) line: 495 FeaturesServiceImpl.installFeature(Feature, EnumSet<Option>) line: 431 FeaturesServiceImpl.installFeature(String, String, EnumSet<Option>) line: 420 Proxyb7a59f09_5bd1_4db1_9436_b2f6b4e1402b.installFeature(String, String, EnumSet) line: not available InstallFeatureCommand.doExecute(FeaturesService) line: 62 InstallFeatureCommand(FeaturesCommandSupport).doExecute() line: 38 InstallFeatureCommand(AbstractAction).execute(CommandSession) line: 33 InstallFeatureCommand(OsgiCommandSupport).execute(CommandSession) line: 39 BlueprintCommand(AbstractCommand).execute(CommandSession, List<Object>) line: 33 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 597 ProxyHandler$1.invoke(Object, Method, Object[]) line: 54 ProxyHandler.invoke(Object, Method, Object[]) line: 119 $BlueprintCommand25912032.execute(CommandSession, List<Object>) line: not available CommandProxy.execute(CommandSession, List<Object>) line: 78 Closure.executeCmd(String, List<Object>) line: 477 Closure.executeStatement(List<Token>) line: 403 Pipe.run() line: 108 Closure.execute(List<Object>) line: 183 Closure.execute(CommandSession, List<Object>) line: 120 CommandSessionImpl.execute(CharSequence) line: 89 ConsoleImpl.run() line: 179 ConsoleFactoryService$3(Thread).run() line: 662 ConsoleFactoryService$3.doRun() line: 115 ConsoleFactoryService$3$1.run() line: 106 AccessController.doPrivileged(PrivilegedAction<T>, AccessControlContext) line: not available [native method] Subject.doAs(Subject, PrivilegedAction<T>) line: 337 ConsoleFactoryService$3.run() line: 104
          Hide
          chris@die-schneider.net Christian Schneider added a comment -

          When setting the log level in karaf to DEBUG we see the loop in waitForInit

          (serverController: true. isInitialized: false)
          2012-08-13 16:36:53,819 | DEBUG | Thread-8 | HttpServiceStarted | 84 - org.ops4j.pax.web.pax-web-runtime - 2.0.2 | isInitialized called for servlet: org.apache.felix.webconsole.internal.KarafOsgiManager@cc8237. Return: false
          2012-08-13 16:36:53,819 | DEBUG | Thread-8 | HttpServiceStarted | 84 - org.ops4j.pax.web.pax-web-runtime - 2.0.2 | isInitialized called for servlet: org.apache.felix.webconsole.internal.KarafOsgiManager@cc8237. Return: false
          2012-08-13 16:36:53,819 | DEBUG | Thread-8 | HttpServiceStarted | 84 - org.ops4j.pax.web.pax-web-runtime - 2.0.2 | Waiting a bit to see if servlet gets initialized (serverController: true. isInitialized: false)

          Show
          chris@die-schneider.net Christian Schneider added a comment - When setting the log level in karaf to DEBUG we see the loop in waitForInit (serverController: true. isInitialized: false) 2012-08-13 16:36:53,819 | DEBUG | Thread-8 | HttpServiceStarted | 84 - org.ops4j.pax.web.pax-web-runtime - 2.0.2 | isInitialized called for servlet: org.apache.felix.webconsole.internal.KarafOsgiManager@cc8237. Return: false 2012-08-13 16:36:53,819 | DEBUG | Thread-8 | HttpServiceStarted | 84 - org.ops4j.pax.web.pax-web-runtime - 2.0.2 | isInitialized called for servlet: org.apache.felix.webconsole.internal.KarafOsgiManager@cc8237. Return: false 2012-08-13 16:36:53,819 | DEBUG | Thread-8 | HttpServiceStarted | 84 - org.ops4j.pax.web.pax-web-runtime - 2.0.2 | Waiting a bit to see if servlet gets initialized (serverController: true. isInitialized: false)
          Hide
          chris@die-schneider.net Christian Schneider added a comment -

          I found a simple workaround. If you first install the http feature and then the webconsole it works. I think the issue might be that the webconsole servlet starts while the pax web config is installed which triggers an update of the ManagedService in pax web. This update restarts the servlet controler and there the block seems to happen.

          Show
          chris@die-schneider.net Christian Schneider added a comment - I found a simple workaround. If you first install the http feature and then the webconsole it works. I think the issue might be that the webconsole servlet starts while the pax web config is installed which triggers an update of the ManagedService in pax web. This update restarts the servlet controler and there the block seems to happen.
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          FYI, we have a similar behavior on 2.2.x using Cellar (depending of the order how you install webconsole, http, and cellar-webconsole features).

          Show
          jbonofre Jean-Baptiste Onofré added a comment - FYI, we have a similar behavior on 2.2.x using Cellar (depending of the order how you install webconsole, http, and cellar-webconsole features).
          Hide
          tjsnell Tracy Snell added a comment -

          I just filed a related or same bug. Easy to reproduce this error.
          KARAF-1901

          Show
          tjsnell Tracy Snell added a comment - I just filed a related or same bug. Easy to reproduce this error. KARAF-1901
          Hide
          andrei.pozolotin Andrei Pozolotin added a comment -

          I also see this with 3.0.0-SNAPSHOT

          Show
          andrei.pozolotin Andrei Pozolotin added a comment - I also see this with 3.0.0-SNAPSHOT
          Hide
          gnt Guillaume Nodet added a comment -

          Log file of the problem

          Show
          gnt Guillaume Nodet added a comment - Log file of the problem
          Hide
          gnt Guillaume Nodet added a comment -

          It seems the problem is caused by pax-web being reconfigured while the webconsole is registering servlets.
          Given pax-web does not unregister and re-register the http service, the console is not aware of the change.

          Show
          gnt Guillaume Nodet added a comment - It seems the problem is caused by pax-web being reconfigured while the webconsole is registering servlets. Given pax-web does not unregister and re-register the http service, the console is not aware of the change.
          Hide
          gnt Guillaume Nodet added a comment -

          Fixed in PAXWEB-448 / PAXWEB-449
          We now just need a release and upgrade to it.

          Show
          gnt Guillaume Nodet added a comment - Fixed in PAXWEB-448 / PAXWEB-449 We now just need a release and upgrade to it.
          Hide
          achim_nierbeck Achim Nierbeck added a comment -

          upgraded to pax-web 1.1.9

          Show
          achim_nierbeck Achim Nierbeck added a comment - upgraded to pax-web 1.1.9

            People

            • Assignee:
              gnt Guillaume Nodet
              Reporter:
              iocanel Ioannis Canellos
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development