Uploaded image for project: 'Karaf'
  1. Karaf
  2. KARAF-1724

Blueprint bundle deadlocks in "Stopping" status when update-ing/dev:watch bundle

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: karaf-osgi
    • Labels:
      None
    • Environment:

      Java Virtual Machine OpenJDK 64-Bit Server VM version 22.0-b10
      Version 1.7.0_03
      org.apache.aries.blueprint (0.3.2)

      Description

      I'm not sure if this should be reported in Karaf or Aries Blueprint, please let me know if I should move it.

      I use "dev:watch *" to update bundles during development. However Karaf usually is unable to stop bundles/services properly during update, Blueprint status for that bundle is stuck in "Stopping"

      By attaching JPDA, it seems that the Blueprint thread is stuck here at Felix.acquireGlobalLock() :

      Daemon Thread [Blueprint Extender: 3] (Suspended)	
      	Object.wait(long) line: not available [native method]	
      	Object[](Object).wait() line: 503	
      	Felix.acquireGlobalLock() line: 4944	
      	StatefulResolver.resolve(Set, Set) line: 102	
      	Felix.resolveBundleRevision(BundleRevision) line: 3819	
      	Felix.getBundleResource(BundleImpl, String) line: 1498	
      	BundleImpl.getResource(String) line: 628	
      	NamespaceHandlerRegistryImpl$NamespaceHandlerSetImpl.<init>(Set<URI>, Bundle) line: 468	
      	NamespaceHandlerRegistryImpl.getNamespaceHandlers(Set<URI>, Bundle) line: 232	
      	BlueprintContainerImpl.doRun() line: 256	
      	BlueprintContainerImpl.run() line: 230	
      	Executors$RunnableAdapter<T>.call() line: 471	
      	FutureTask$Sync.innerRun() line: 334	
      	ScheduledThreadPoolExecutor$ScheduledFutureTask<V>(FutureTask<V>).run() line: 166	
      	ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.access$201(ScheduledThreadPoolExecutor$ScheduledFutureTask) line: 178	
      	ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.run() line: 292	
      	ScheduledThreadPoolExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: 1110	
      	ThreadPoolExecutor$Worker.run() line: 603	
      	Thread.run() line: 722	
      

      While other Blueprint threads seem "normal":

      Daemon Thread [Blueprint Extender: 1] (Suspended)	
      	Unsafe.park(boolean, long) line: not available [native method]	
      	LockSupport.parkNanos(Object, long) line: 226	
      	AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) line: 2082	
      	ScheduledThreadPoolExecutor$DelayedWorkQueue.take() line: 1090	
      	ScheduledThreadPoolExecutor$DelayedWorkQueue.take() line: 807	
      	ScheduledThreadPoolExecutor(ThreadPoolExecutor).getTask() line: 1043	
      	ScheduledThreadPoolExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: 1103	
      	ThreadPoolExecutor$Worker.run() line: 603	
      	Thread.run() line: 722	
      

      List of bundles:

      
      START LEVEL 100 , List Threshold: 0
         ID   State         Blueprint      Level  Symbolic name
      [   0] [Active     ] [            ] [    0] org.apache.felix.framework (4.0.3)
      [   1] [Active     ] [            ] [    5] org.ops4j.pax.url.mvn (1.3.5)
      [   2] [Active     ] [            ] [    5] org.ops4j.pax.url.wrap (1.3.5)
      [   3] [Active     ] [            ] [    8] org.ops4j.pax.logging.pax-logging-service (1.6.9)
      [   4] [Active     ] [            ] [    8] org.ops4j.pax.logging.pax-logging-api (1.6.9)
      [   5] [Active     ] [            ] [   10] org.apache.felix.configadmin (1.4.0)
      [   6] [Active     ] [            ] [   11] org.apache.felix.fileinstall (3.2.4)
      [   7] [Active     ] [            ] [   20] org.apache.aries.proxy (0.3.1)
      [   8] [Active     ] [            ] [   20] org.apache.servicemix.bundles.asm (3.3.1.1)
      [   9] [Active     ] [Created     ] [   20] org.apache.aries.blueprint (0.3.2)
      [  10] [Active     ] [            ] [   20] org.apache.aries.util (0.3.1)
      [  11] [Active     ] [Created     ] [   28] org.apache.karaf.deployer.blueprint (2.3.0.SNAPSHOT)
      [  12] [Active     ] [Created     ] [   28] org.apache.karaf.deployer.spring (2.3.0.SNAPSHOT)
      [  13] [Active     ] [Created     ] [   30] org.apache.karaf.deployer.features (2.3.0.SNAPSHOT)
      [  14] [Active     ] [            ] [   30] org.apache.karaf.diagnostic.core (2.3.0.SNAPSHOT)
      [  15] [Active     ] [Created     ] [   30] org.apache.karaf.shell.commands (2.3.0.SNAPSHOT)
      [  16] [Active     ] [Created     ] [   30] org.apache.karaf.features.core (2.3.0.SNAPSHOT)
      [  17] [Active     ] [Created     ] [   30] org.apache.karaf.diagnostic.command (2.3.0.SNAPSHOT)
      [  18] [Active     ] [Created     ] [   30] org.apache.karaf.deployer.wrap (2.3.0.SNAPSHOT)
      [  19] [Active     ] [Created     ] [   30] org.apache.karaf.jaas.modules (2.3.0.SNAPSHOT)
      [  20] [Active     ] [Created     ] [   30] org.apache.karaf.admin.core (2.3.0.SNAPSHOT)
      [  21] [Active     ] [Created     ] [   30] org.apache.karaf.shell.osgi (2.3.0.SNAPSHOT)
      [  22] [Active     ] [            ] [   30] org.apache.sshd.core (0.7.0)
      [  23] [Active     ] [            ] [   30] org.apache.aries.jmx.blueprint (0.3.0)
      [  24] [Active     ] [Created     ] [   30] org.apache.karaf.shell.dev (2.3.0.SNAPSHOT)
      [  25] [Active     ] [Created     ] [   30] org.apache.karaf.diagnostic.common (2.3.0.SNAPSHOT)
      [  26] [Active     ] [Created     ] [   30] org.apache.karaf.diagnostic.management (2.3.0.SNAPSHOT)
      [  27] [Active     ] [Created     ] [   30] org.apache.karaf.shell.console (2.3.0.SNAPSHOT)
      [  28] [Active     ] [            ] [   30] org.apache.aries.jmx.api (0.3.0)
      [  29] [Active     ] [Created     ] [   30] org.apache.karaf.deployer.kar (2.3.0.SNAPSHOT)
      [  30] [Active     ] [Created     ] [   30] org.apache.karaf.features.management (2.3.0.SNAPSHOT)
      [  31] [Active     ] [Created     ] [   30] org.apache.karaf.features.command (2.3.0.SNAPSHOT)
      [  32] [Active     ] [Created     ] [   30] org.apache.karaf.shell.log (2.3.0.SNAPSHOT)
      [  33] [Active     ] [            ] [   30] org.apache.mina.core (2.0.4)
      [  34] [Active     ] [Created     ] [   30] org.apache.karaf.admin.management (2.3.0.SNAPSHOT)
      [  35] [Active     ] [Created     ] [   30] org.apache.karaf.management.server (2.3.0.SNAPSHOT)
      [  36] [Active     ] [Created     ] [   30] org.apache.karaf.shell.packages (2.3.0.SNAPSHOT)
      [  37] [Active     ] [Created     ] [   30] org.apache.karaf.shell.ssh (2.3.0.SNAPSHOT)
      [  38] [Active     ] [Created     ] [   30] org.apache.karaf.admin.command (2.3.0.SNAPSHOT)
      [  39] [Active     ] [Created     ] [   30] org.apache.karaf.jaas.command (2.3.0.SNAPSHOT)
      [  40] [Active     ] [            ] [   30] org.apache.aries.jmx.core (0.3.1)
      [  41] [Active     ] [Created     ] [   30] org.apache.karaf.jaas.config (2.3.0.SNAPSHOT)
      [  42] [Active     ] [Created     ] [   30] org.apache.karaf.shell.config (2.3.0.SNAPSHOT)
      [  43] [Active     ] [Created     ] [   30] org.apache.karaf.management.mbeans.system (2.3.0.SNAPSHOT)
      [  44] [Active     ] [Created     ] [   30] org.apache.karaf.management.mbeans.bundles (2.3.0.SNAPSHOT)
      [  45] [Active     ] [Created     ] [   30] org.apache.karaf.management.mbeans.services (2.3.0.SNAPSHOT)
      [  46] [Active     ] [Created     ] [   30] org.apache.karaf.management.mbeans.config (2.3.0.SNAPSHOT)
      [  47] [Active     ] [Created     ] [   30] org.apache.karaf.management.mbeans.log (2.3.0.SNAPSHOT)
      [  48] [Active     ] [Created     ] [   30] org.apache.karaf.management.mbeans.packages (2.3.0.SNAPSHOT)
      [  49] [Active     ] [Created     ] [   30] org.apache.karaf.management.mbeans.dev (2.3.0.SNAPSHOT)
      [  50] [Active     ] [            ] [   50] org.apache.ws.commons.axiom.axiom-dom (1.2.13)
      [ 115] [Active     ] [            ] [   50] org.apache.ws.commons.axiom.axiom-impl (1.2.13)
      [ 588] [Active     ] [            ] [   40] org.apache.geronimo.specs.geronimo-annotation_1.1_spec (1.0.1)
      [ 589] [Active     ] [            ] [   40] com.google.guava (12.0.0)
      [ 590] [Active     ] [            ] [   40] org.apache.commons.io (2.4.0)
      [ 591] [Active     ] [            ] [   40] org.apache.commons.lang3 (3.1.0)
      [ 592] [Active     ] [            ] [   40] org.apache.servicemix.bundles.xpp3 (1.1.0.4c_6)
      [ 593] [Active     ] [            ] [   40] org.apache.servicemix.bundles.xstream (1.4.1.1)
      [ 594] [Active     ] [            ] [   40] jackson-core (2.0.4)
      [ 595] [Active     ] [            ] [   40] jackson-annotations (2.0.4)
      [ 596] [Active     ] [            ] [   40] jackson-databind (2.0.4)
      [ 597] [Active     ] [            ] [   40] joda-time (2.1)
      [ 598] [Active     ] [            ] [   40] jackson-datatype-joda (2.0.4)
      [ 599] [Active     ] [            ] [   40] jackson-datatype-guava (2.0.3)
      [ 600] [Active     ] [            ] [   50] org.apache.httpcomponents.httpcore (4.2.1)
      [ 601] [Active     ] [            ] [   50] org.apache.httpcomponents.httpclient (4.2.1)
      [ 602] [Active     ] [            ] [   30] org.apache.geronimo.specs.geronimo-jta_1.1_spec (1.1.1)
      [ 603] [Active     ] [            ] [   30] org.apache.geronimo.specs.geronimo-jpa_2.0_spec (1.1)
      [ 604] [Active     ] [            ] [   30] org.apache.aries.jpa.api (0.3.0)
      [ 605] [Active     ] [Created     ] [   30] org.apache.aries.jpa.blueprint.aries (0.3.0)
      [ 606] [Active     ] [            ] [   30] org.apache.aries.jpa.container (0.3.0)
      [ 607] [Active     ] [            ] [   30] org.apache.aries.jpa.container.context (0.3.0)
      [ 608] [Active     ] [            ] [   10] org.apache.servicemix.specs.activation-api-1.1 (1.9.0)
      [ 609] [Active     ] [            ] [   10] org.apache.servicemix.specs.stax-api-1.0 (1.9.0)
      [ 610] [Active     ] [            ] [   10] org.apache.servicemix.specs.jaxb-api-2.2 (1.9.0)
      [ 611] [Active     ] [            ] [   10] stax2-api (3.1.1)
      [ 612] [Active     ] [            ] [   10] woodstox-core-asl (4.1.2)
      [ 613] [Active     ] [            ] [   10] org.apache.servicemix.bundles.jaxb-impl (2.2.1.1_2)
      [ 614] [Active     ] [            ] [   50] soluvas-commons (1.0.0.SNAPSHOT)
      [ 615] [Active     ] [Created     ] [   50] soluvas-json (1.0.0.SNAPSHOT)
      [ 616] [Active     ] [            ] [   50] soluvas-cdi (1.0.0.SNAPSHOT)
      [ 617] [Active     ] [            ] [   50] soluvas-jaxrs (1.0.0.SNAPSHOT)
      [ 618] [Active     ] [Created     ] [   30] org.apache.aries.transaction.blueprint (0.3.0)
      [ 619] [Active     ] [            ] [   30] org.apache.aries.transaction.manager (0.3.1)
      [ 620] [Active     ] [            ] [   30] org.apache.aries.transaction.wrappers (0.3.0)
      [ 621] [Active     ] [            ] [   30] org.apache.servicemix.specs.activation-api-1.1 (2.0.0)
      [ 622] [Active     ] [            ] [   30] org.apache.geronimo.specs.geronimo-servlet_2.5_spec (1.1.2)
      [ 623] [Active     ] [            ] [   30] javax.mail (1.4.4)
      [ 624] [Active     ] [            ] [   30] org.eclipse.jetty.util (7.6.2.v20120308)
      [ 625] [Active     ] [            ] [   30] org.eclipse.jetty.io (7.6.2.v20120308)
      [ 626] [Active     ] [            ] [   30] org.eclipse.jetty.http (7.6.2.v20120308)
      [ 627] [Active     ] [            ] [   30] org.eclipse.jetty.client (7.6.2.v20120308)
      [ 628] [Active     ] [            ] [   30] org.eclipse.jetty.continuation (7.6.2.v20120308)
      [ 629] [Active     ] [            ] [   30] org.eclipse.jetty.jmx (7.6.2.v20120308)
      [ 630] [Active     ] [            ] [   30] org.eclipse.jetty.server (7.6.2.v20120308)
      [ 631] [Active     ] [            ] [   30] org.eclipse.jetty.security (7.6.2.v20120308)
      [ 632] [Active     ] [            ] [   30] org.eclipse.jetty.servlet (7.6.2.v20120308)
      [ 633] [Active     ] [            ] [   30] org.eclipse.jetty.servlets (7.6.2.v20120308)
      [ 634] [Active     ] [            ] [   30] org.eclipse.jetty.xml (7.6.2.v20120308)
      [ 635] [Active     ] [            ] [   30] org.eclipse.jetty.webapp (7.6.2.v20120308)
      [ 636] [Active     ] [            ] [   30] org.eclipse.jetty.jndi (7.6.2.v20120308)
      [ 637] [Active     ] [            ] [   30] org.eclipse.jetty.plus (7.6.2.v20120308)
      [ 638] [Active     ] [            ] [   30] org.eclipse.jetty.websocket (7.6.2.v20120308)
      [ 639] [Active     ] [            ] [   30] org.ops4j.pax.web.pax-web-api (1.1.3)
      [ 640] [Active     ] [            ] [   30] org.ops4j.pax.web.pax-web-spi (1.1.3)
      [ 641] [Active     ] [            ] [   30] org.ops4j.pax.web.pax-web-runtime (1.1.3)
      [ 642] [Active     ] [            ] [   30] org.ops4j.pax.web.pax-web-jetty (1.1.3)
      [ 643] [Active     ] [            ] [   50] org.apache.geronimo.specs.geronimo-ws-metadata_2.0_spec (1.1.3)
      [ 644] [Active     ] [            ] [   50] org.apache.geronimo.specs.geronimo-jms_1.1_spec (1.1.1)
      [ 645] [Active     ] [            ] [   50] org.apache.servicemix.specs.jsr311-api-1.1.1 (2.0.0)
      [ 646] [Active     ] [            ] [   50] org.apache.commons.codec (1.4)
      [ 647] [Active     ] [            ] [   50] org.apache.servicemix.bundles.commons-httpclient (3.1.0.7)
      [ 648] [Active     ] [            ] [   50] org.apache.servicemix.bundles.stax-utils (20040917.0.0.2)
      [ 649] [Active     ] [            ] [   50] org.apache.servicemix.bundles.wsdl4j (1.6.2.6)
      [ 650] [Active     ] [            ] [   50] org.apache.james.apache-mime4j-core (0.7.2)
      [ 651] [Active     ] [            ] [   50] org.apache.ws.commons.axiom.axiom-api (1.2.13)
      [ 652] [Active     ] [            ] [   50] org.apache.commons.fileupload (1.2.2)
      [ 653] [Active     ] [            ] [   50] org.apache.neethi (3.0.2)
      [ 654] [Active     ] [            ] [   50] org.apache.servicemix.bundles.xmlschema (1.4.3.1)
      [ 655] [Active     ] [            ] [   50] org.apache.woden.woden-api (1.0.0)
      [ 656] [Active     ] [            ] [   50] org.apache.woden.woden-impl-commons (1.0.0)
      [ 657] [Active     ] [            ] [   50] org.apache.woden.woden-impl-dom (1.0.0)
      [ 659] [Active     ] [            ] [   50] org.apache.servicemix.bundles.commons-beanutils (1.8.3.1)
      [ 660] [Active     ] [            ] [   50] org.apache.servicemix.bundles.rhino (1.0.0.7R2_3)
      [ 661] [Active     ] [            ] [   60] com.google.code.magja (1.0.3.SNAPSHOT)
      [ 662] [Active     ] [            ] [   50] org.apache.commons.pool (1.6.0)
      [ 663] [Active     ] [            ] [   50] org.apache.servicemix.bundles.antlr (2.7.7.5)
      [ 664] [Active     ] [            ] [   50] org.apache.servicemix.bundles.commons-collections (3.2.1.3)
      [ 665] [Active     ] [            ] [   50] org.apache.commons.lang (2.6)
      [ 666] [Active     ] [            ] [   50] org.apache.directory.shared.ldap.client.api (1.0.0.M12)
      [ 667] [Active     ] [            ] [   50] org.apache.directory.shared.asn1.api (1.0.0.M12)
      [ 668] [Active     ] [            ] [   50] org.apache.directory.shared.asn1.ber (1.0.0.M12)
      [ 669] [Active     ] [            ] [   50] org.apache.directory.shared.i18n (1.0.0.M12)
      [ 670] [Active     ] [            ] [   50] org.apache.directory.shared.util (1.0.0.M12)
      [ 671] [Active     ] [            ] [   50] org.apache.directory.shared.ldap.codec.core (1.0.0.M12)
      [ 672] [Active     ] [            ] [   50] org.apache.directory.shared.ldap.net.mina (1.0.0.M12)
      [ 673] [Active     ] [            ] [   50] org.apache.directory.shared.ldap.model (1.0.0.M12)
      [ 674] [Active     ] [            ] [   50] org.apache.directory.shared.ldap.schema (1.0.0.M12)
      [ 675] [Active     ] [            ] [   50] org.apache.directory.shared.ldap.extras.aci (1.0.0.M12)
      [ 676] [Active     ] [            ] [   50] org.apache.servicemix.bundles.jaxb-xjc (2.2.1.1_2)
      [ 677] [Active     ] [            ] [   50] wrap_mvn_com.google.protobuf_protobuf-java_2.4.1_Export-Package___version_2.4.1 (0)
      [ 678] [Active     ] [            ] [   50] org.mvel2 (2.1.0.drools16)
      [ 679] [Active     ] [            ] [   50] wrap_mvn_com.sun.istack_istack-commons-runtime_2.12_Export-Package___version_2.12.0 (0)
      [ 680] [Active     ] [            ] [   50] wrap_mvn_com.thoughtworks.xstream_xstream_1.4.2_Export-Package___version_1.4.2 (0)
      [ 681] [Active     ] [            ] [   50] org.apache.servicemix.bundles.antlr-runtime (3.4.0.2)
      [ 682] [Active     ] [            ] [   50] org.drools.core (5.4.0.Final)
      [ 683] [Active     ] [            ] [   50] org.drools.compiler (5.4.0.Final)
      [ 684] [Active     ] [            ] [   50] org.drools.templates (5.4.0.Final)
      [ 685] [Active     ] [            ] [   50] org.drools.api (5.4.0.Final)
      [ 686] [Active     ] [            ] [   50] org.drools.internalapi (5.4.0.Final)
      [ 687] [Active     ] [            ] [   50] org.jbpm.flow.core (5.3.0.Final)
      [ 688] [Active     ] [            ] [   50] org.jbpm.flow.builder (5.3.0.Final)
      [ 689] [Active     ] [            ] [   50] org.jbpm.bpmn2 (5.3.0.Final)
      [ 690] [Resolved   ] [            ] [   50] wrap_mvn_org.neo4j_neo4j-kernel_1.7.2 (0)
      [ 691] [Resolved   ] [            ] [   50] wrap_mvn_org.neo4j_neo4j-lucene-index_1.7.2 (0)
      [ 692] [Resolved   ] [            ] [   50] wrap_mvn_org.neo4j_neo4j-cypher_1.7.2 (0)
      [ 693] [Resolved   ] [            ] [   50] wrap_mvn_org.neo4j_neo4j-rest-graphdb_1.7 (0)
      [ 694] [Active     ] [            ] [   50] wrap_mvn_com.tinkerpop.blueprints_blueprints-core_1.2 (0)
      [ 695] [Active     ] [            ] [   50] wrap_mvn_com.tinkerpop.blueprints_blueprints-neo4j-graph_1.2 (0)
      [ 696] [Active     ] [            ] [   50] wrap_mvn_com.tinkerpop_frames_0.7 (0)
      [ 697] [Active     ] [            ] [   50] wrap_mvn_net.coobird_thumbnailator_0.4.1 (0)
      [ 698] [Active     ] [            ] [   50] soluvas-image-store (1.0.0.SNAPSHOT)
      [ 699] [Active     ] [            ] [   50] org.apache.geronimo.specs.geronimo-validation_1.0_spec (1.1)
      [ 700] [Active     ] [            ] [   50] wrap_mvn_org.picketlink.idm_picketlink-idm-common_1.5.0.Alpha02_Export-Package___version_1.5.0.Alpha02 (0)
      [ 701] [Active     ] [            ] [   50] wrap_mvn_org.picketlink.idm_picketlink-idm-api_1.5.0.Alpha02_Export-Package___version_1.5.0.Alpha02 (0)
      [ 702] [Active     ] [            ] [   50] soluvas-ldap (1.0.0.SNAPSHOT)
      [ 703] [Active     ] [            ] [   50] org.apache.geronimo.specs.geronimo-ejb_3.1_spec (1.0.2)
      [ 704] [Active     ] [            ] [   50] org.apache.geronimo.specs.geronimo-jaxrpc_1.1_spec (2.0.0)
      [ 705] [Active     ] [            ] [   50] javax.cache (0.4.0)
      [ 706] [Active     ] [            ] [   50] org.apache.geronimo.specs.geronimo-jcdi_1.0_spec (1.0)
      [ 707] [Active     ] [            ] [   50] javax.el-api (2.2.99.b01)
      [ 708] [Active     ] [            ] [   50] org.apache.geronimo.specs.geronimo-atinject_1.0_spec (1.0)
      [ 709] [Active     ] [            ] [   50] org.apache.geronimo.specs.geronimo-interceptor_3.0_spec (1.0.1)
      [ 711] [Active     ] [            ] [   50] com.typesafe.config (0.4.1)
      [ 713] [Active     ] [            ] [   50] wrap_mvn_com.github.spullara.mustache.java_compiler_0.8.2 (0)
      [ 715] [Active     ] [            ] [   50] wrap_mvn_com.google.code.morphia_morphia-logging-slf4j_0.99 (0)
      [ 716] [Active     ] [            ] [   50] org.mongodb.mongo-java-driver (2.8.0.RELEASE)
      [ 717] [Active     ] [            ] [   50] com.rabbitmq.client (2.8.5)
      [ 718] [Active     ] [            ] [   50] wrap_mvn_com.restfb_restfb_1.6.9 (0)
      [ 719] [Active     ] [            ] [   50] org.apache.commons.email (1.2)
      [ 720] [Active     ] [            ] [   50] jackson-core-asl (1.9.9)
      [ 721] [Active     ] [            ] [   50] jackson-mapper-asl (1.9.9)
      [ 722] [Active     ] [            ] [   50] joda-time (1.6.2)
      [ 723] [Active     ] [GracePeriod ] [  100] id.co.bippo.commerce.core (4.0.1.SNAPSHOT)
      [ 724] [Active     ] [            ] [   50] wrap_mvn_org.apache.jena_jena-iri_0.9.3 (0)
      [ 725] [Active     ] [            ] [   50] wrap_mvn_org.apache.jena_jena-core_2.7.3 (0)
      [ 726] [Active     ] [            ] [   50] wrap_mvn_org.apache.jena_jena-arq_2.9.3 (0)
      [ 728] [Active     ] [            ] [   55] org.apache.axis2.osgi (1.0.0.2)
      [ 729] [Stopping   ] [            ] [   80] id.co.bippo.commerce.shell (4.0.1.SNAPSHOT)
      [ 732] [Active     ] [            ] [   80] com.typesafe.akka.actor (2.1.0.M1)
      [ 733] [Active     ] [            ] [   80] org.scala-lang-osgi.scala-library (2.10.0.M6)
      [ 743] [Installed  ] [            ] [   80] org.eclipse.emf.common (2.8.0.v20120606-0717)
      [ 744] [Installed  ] [            ] [   80] org.eclipse.emf.ecore.xmi (2.8.0.v20120606-0717)
      [ 745] [Installed  ] [            ] [   80] org.eclipse.emf.ecore (2.8.0.v20120606-0717)
      [ 747] [Resolved   ] [            ] [   80] wrap_mvn_com.google.code.morphia_morphia_0.99_DynamicImport-Package__ (0)
      [ 749] [Resolved   ] [            ] [   80] com.mysql.jdbc (5.1.20)
      [ 750] [Active     ] [Created     ] [   80] berbatik_dev_magento_mysql.xml (0.0.0)
      [ 751] [Active     ] [            ] [   30] org.apache.aries.jndi.api (0.3.0)
      [ 752] [Active     ] [            ] [   30] org.apache.aries.jndi.core (0.3.0)
      [ 753] [Active     ] [            ] [   30] org.apache.aries.jndi.rmi (0.3.0)
      [ 754] [Active     ] [            ] [   30] org.apache.aries.jndi.url (0.3.0)
      [ 755] [Active     ] [            ] [   30] org.apache.aries.jndi.legacy.support (0.3.0)
      

      The affected bundle:

      [ 729] [Stopping   ] [            ] [   80] id.co.bippo.commerce.shell (4.0.1.SNAPSHOT)
      

      Headers:

      
      karaf@root> headers 729
      
      Bippo Commerce Shell (729)
      --------------------------
      Manifest-Version = 1.0
      Bnd-LastModified = 1344892875353
      Tool = Bnd-1.50.0
      Built-By = ceefour
      Build-Jdk = 1.6.0_24
      Created-By = Apache Maven Bundle Plugin
      
      Bundle-Vendor = Bippo Indonesia
      Bundle-Name = Bippo Commerce Shell
      Bundle-DocURL = http://www.bippo.co.id/
      Bundle-Description = Shell commands for Product management, Booking management
      Bundle-SymbolicName = id.co.bippo.commerce.shell
      Bundle-Version = 4.0.1.SNAPSHOT
      Bundle-License = http://www.bippo.co.id/
      Bundle-ManifestVersion = 2
      
      Import-Package = 
              com.google.code.magja.model.product;version="[1.0,2)",
              com.google.code.magja.service;version="[1.0,2)",
              com.google.code.magja.service.product;version="[1.0,2)",
              com.google.code.magja.soap;version="[1.0,2)",
              com.google.common.base;version="[12.0,13)",
              com.google.common.collect;version="[12.0,13)",
              com.google.common.util.concurrent;version="[12.0,13)",
              com.hp.hpl.jena.rdf.model,
              com.hp.hpl.jena.shared,
              id.co.bippo.commerce.core;version="[4.0,5)",
              id.co.bippo.commerce.dao;version="[4.0,5)",
              javax.annotation,
              javax.inject,
              org.apache.commons.beanutils;version="[1.8,2)",
              org.apache.directory.ldap.client.api;version="[1.0,2)",
              org.apache.felix.gogo.commands;version="[0.10,1)",
              org.apache.felix.service.command;status=provisional;version="[0.10,1)",
              org.apache.karaf.shell.console;version="[2.2,3)",
              org.codehaus.jackson.map;version="[1.9,2)",
              org.osgi.framework;version="[1.5,2)",
              org.osgi.service.blueprint;version="[1.0.0,2.0.0)",
              org.osgi.util.tracker;version="[1.4,2)",
              org.slf4j;version="[1.6,2)",
              org.soluvas.image.store;version="[1.0,2)",
              org.soluvas.ldap;version="[1.0,2)"
      Export-Package = 
              com.soluvas.multitenant;
                      uses:="org.apache.felix.service.command,
                              com.google.common.util.concurrent,
                              com.google.common.base,
                              javax.annotation,
                              org.osgi.util.tracker,
                              org.osgi.framework,
                              org.slf4j";
                      version=4.0.1.SNAPSHOT,
              com.soluvas.person.shell;
                      uses:="org.apache.karaf.shell.console,
                              com.google.common.collect,
                              org.soluvas.ldap,
                              org.apache.felix.gogo.commands,
                              org.slf4j";
                      version=4.0.1.SNAPSHOT,
              id.co.bippo.commerce.core.shell;
                      uses:="id.co.bippo.commerce.core,
                              javax.annotation,
                              org.soluvas.ldap,
                              com.google.common.base,
                              org.apache.felix.service.command,
                              org.apache.karaf.shell.console,
                              com.soluvas.multitenant,
                              org.apache.felix.gogo.commands,
                              org.slf4j";
                      version=4.0.1.SNAPSHOT,
              id.co.bippo.commerce.dao.shell;
                      uses:="javax.annotation,
                              com.google.common.base,
                              id.co.bippo.commerce.dao,
                              org.apache.felix.service.command,
                              org.apache.karaf.shell.console,
                              com.soluvas.multitenant,
                              org.apache.felix.gogo.commands,
                              org.slf4j";
                      version=4.0.1.SNAPSHOT,
              id.co.bippo.commerce.shell;
                      uses:="javax.annotation,
                              id.co.bippo.commerce.core,
                              com.google.code.magja.model.product,
                              com.google.code.magja.service,
                              com.google.code.magja.service.product,
                              com.google.code.magja.soap,
                              com.hp.hpl.jena.shared,
                              org.slf4j,
                              com.hp.hpl.jena.rdf.model,
                              org.codehaus.jackson.map,
                              javax.inject";
                      version=4.0.1.SNAPSHOT,
              org.soluvas.image.store.shell;
                      uses:="org.soluvas.image.store,
                              javax.annotation,
                              com.google.common.base,
                              org.apache.commons.beanutils,
                              org.apache.felix.service.command,
                              org.apache.karaf.shell.console,
                              com.soluvas.multitenant,
                              org.apache.felix.gogo.commands,
                              org.slf4j";
                      version=4.0.1.SNAPSHOT
      

      I also can't exit Karaf properly, i.e. using Ctrl+D still hangs. I need to Ctrl+Z (sleep), then do:

      kill -9 %1
      

      to forcefully kill karaf.

      I also am unable to run some other commands like dev:show-tree or osgi:install :

      karaf@root> dev:show-tree 729
      Bundle id.co.bippo.commerce.shell [729] is currently STOPPING
      (..........hangs here forever - slept for 6 hours and woke up - still at the same spot.......)
      ^C
      Error executing command: Unable to acquire global lock to retrieve exported packages.
      

      log:

      2012-08-14 10:44:48,592 | INFO  | l Console Thread | Console                          | araf.shell.console.jline.Console  196 | 27 - org.apache.karaf.shell.console - 2.3.0.SNAPSHOT | Exception caught while executing command
      org.apache.karaf.shell.console.MultiException: Error installing bundles:
              Unable to install bundle mvn:org.apache.openjpa/openjpa/2.2.0
              at org.apache.karaf.shell.console.MultiException.throwIf(MultiException.java:91)
              at org.apache.karaf.shell.osgi.InstallBundle.doExecute(InstallBundle.java:70)[21:org.apache.karaf.shell.osgi:2.3.0.SNAPSHOT]
              at org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:474)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:400)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:89)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.karaf.shell.console.jline.Console.run(Console.java:167)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at java.lang.Thread.run(Thread.java:722)[:1.7.0_03]
      Caused by: java.lang.Exception: Unable to install bundle mvn:org.apache.openjpa/openjpa/2.2.0
              at org.apache.karaf.shell.osgi.InstallBundle.doExecute(InstallBundle.java:45)[21:org.apache.karaf.shell.osgi:2.3.0.SNAPSHOT]
              ... 11 more
      Caused by: org.osgi.framework.BundleException: Unable to acquire the global lock to install the bundle.
              at org.apache.felix.framework.Felix.installBundle(Felix.java:2727)[org.apache.felix.framework-4.0.3.jar:]
              at org.apache.felix.framework.BundleContextImpl.installBundle(BundleContextImpl.java:165)[org.apache.felix.framework-4.0.3.jar:]
              at org.apache.karaf.shell.osgi.InstallBundle.doExecute(InstallBundle.java:43)[21:org.apache.karaf.shell.osgi:2.3.0.SNAPSHOT]
              ... 11 more
      

      install bundle locks up:

      karaf@root> install -s mvn:org.apache.openjpa/openjpa/2.2.0
      (......locks up here forever......)
      ^C
      Bundle IDs: 
      Error executing command: Error installing bundles:
              Unable to install bundle mvn:org.apache.openjpa/openjpa/2.2.0
      

      log:

      2012-08-14 16:01:20,459 | INFO  | l Console Thread | Console                          | araf.shell.console.jline.Console  196 | 27 - org.apache.karaf.shell.console - 2.3.0.SNAPSHOT | Exception caught while executing command
      org.apache.karaf.shell.console.MultiException: Error installing bundles:
              Unable to install bundle mvn:org.apache.openjpa/openjpa/2.2.0
              at org.apache.karaf.shell.console.MultiException.throwIf(MultiException.java:91)
              at org.apache.karaf.shell.osgi.InstallBundle.doExecute(InstallBundle.java:70)[21:org.apache.karaf.shell.osgi:2.3.0.SNAPSHOT]
              at org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:474)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:400)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:89)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at org.apache.karaf.shell.console.jline.Console.run(Console.java:167)[27:org.apache.karaf.shell.console:2.3.0.SNAPSHOT]
              at java.lang.Thread.run(Thread.java:722)[:1.7.0_03]
      Caused by: java.lang.Exception: Unable to install bundle mvn:org.apache.openjpa/openjpa/2.2.0
              at org.apache.karaf.shell.osgi.InstallBundle.doExecute(InstallBundle.java:45)[21:org.apache.karaf.shell.osgi:2.3.0.SNAPSHOT]
              ... 11 more
      Caused by: org.osgi.framework.BundleException: Unable to acquire the global lock to install the bundle.
              at org.apache.felix.framework.Felix.installBundle(Felix.java:2727)[org.apache.felix.framework-4.0.3.jar:]
              at org.apache.felix.framework.BundleContextImpl.installBundle(BundleContextImpl.java:165)[org.apache.felix.framework-4.0.3.jar:]
              at org.apache.karaf.shell.osgi.InstallBundle.doExecute(InstallBundle.java:43)[21:org.apache.karaf.shell.osgi:2.3.0.SNAPSHOT]
              ... 11 more
      

      The strange thing is that these commands do not trigger the bug:

      refresh <bundle>
      update <bundle>
      stop <bundle> then start <bundle>
      

      I've been having this issue quite a lot, I think it's quite a common problem in OSGi world, and I suspect it may have something to do with my bundle.

      Is there a procedure or something to diagnose/debug such problems?

      Or perhaps best practices to avoid this issue ?

      Let me know if you need more information.

        Issue Links

          Activity

          Hide
          ceefour Hendy Irawan added a comment - - edited

          Doing this:

          1. Disable dev:watch
          2. Rebuild the bundle
          3. update <bundle>

          will still cause it to be stuck in "Stopping" state, so it's the same effect as dev:watch :

          [ 206] [Stopping   ] [            ] [   80] Bippo Commerce Shell (4.0.1.SNAPSHOT)
          

          I also found an issue, probably with my own bundles:

          [ 199] [Active     ] [Created     ] [  100] id.co.bippo.commerce.core (4.0.1.SNAPSHOT)
          [ 200] [Active     ] [Created     ] [   50] com.soluvas.data (1.0.0.SNAPSHOT)
          [ 201] [Active     ] [            ] [   50] wrap_mvn_org.apache.jena_jena-iri_0.9.3 (0)
          [ 202] [Active     ] [            ] [   50] wrap_mvn_org.apache.jena_jena-core_2.7.3 (0)
          [ 203] [Active     ] [            ] [   50] wrap_mvn_org.apache.jena_jena-arq_2.9.3 (0)
          [ 206] [Installed  ] [            ] [   80] id.co.bippo.commerce.shell (4.0.1.SNAPSHOT)
          [ 207] [Active     ] [Created     ] [   80] berbatik_dev_magento_mysql.xml (0.0.0)
          [ 208] [Active     ] [            ] [   50] org.eclipse.jdt.core.compiler.batch (3.3.0)
          

          The bundles I updated are 199 and 206, which seems to be updated properly. However bundle 206 won't start. Attempting to start it throws this error:

          karaf@root> start 206
          Error executing command: Error starting bundles:
          lve bundle revision id.co.bippo.commerce.shell [206.6] because it is exposed to package 'id.co.bippo.commerce.dao' from bundle revisions id.co.bippo.commerce.core [199.11] and id.co.bippo.commerce.core [199.12] via two dependency chains.
          
          Chain 1:
            id.co.bippo.commerce.shell [206.6]
              import: (&(osgi.wiring.package=id.co.bippo.commerce.dao)(version>=4.0.0)(!(version>=5.0.0)))
               |
              export: osgi.wiring.package=id.co.bippo.commerce.dao
            id.co.bippo.commerce.core [199.11]
          
          Chain 2:
            id.co.bippo.commerce.shell [206.6]
              import: (&(osgi.wiring.package=id.co.bippo.commerce.core)(version>=4.0.0)(!(version>=5.0.0)))
               |
              export: osgi.wiring.package=id.co.bippo.commerce.core; uses:=id.co.bippo.commerce.dao
              export: osgi.wiring.package=id.co.bippo.commerce.dao
            id.co.bippo.commerce.core [199.12]
          

          which seems to say that bundle 199 was not cleanly unloaded before updated? (so there are two versions 199.11 and 199.12)?

          Doing a:

          refresh 199
          

          makes bundle 206 startable properly.

          Show
          ceefour Hendy Irawan added a comment - - edited Doing this: 1. Disable dev:watch 2. Rebuild the bundle 3. update <bundle> will still cause it to be stuck in "Stopping" state, so it's the same effect as dev:watch : [ 206] [Stopping ] [ ] [ 80] Bippo Commerce Shell (4.0.1.SNAPSHOT) I also found an issue, probably with my own bundles: [ 199] [Active ] [Created ] [ 100] id.co.bippo.commerce.core (4.0.1.SNAPSHOT) [ 200] [Active ] [Created ] [ 50] com.soluvas.data (1.0.0.SNAPSHOT) [ 201] [Active ] [ ] [ 50] wrap_mvn_org.apache.jena_jena-iri_0.9.3 (0) [ 202] [Active ] [ ] [ 50] wrap_mvn_org.apache.jena_jena-core_2.7.3 (0) [ 203] [Active ] [ ] [ 50] wrap_mvn_org.apache.jena_jena-arq_2.9.3 (0) [ 206] [Installed ] [ ] [ 80] id.co.bippo.commerce.shell (4.0.1.SNAPSHOT) [ 207] [Active ] [Created ] [ 80] berbatik_dev_magento_mysql.xml (0.0.0) [ 208] [Active ] [ ] [ 50] org.eclipse.jdt.core.compiler.batch (3.3.0) The bundles I updated are 199 and 206, which seems to be updated properly. However bundle 206 won't start. Attempting to start it throws this error: karaf@root> start 206 Error executing command: Error starting bundles: lve bundle revision id.co.bippo.commerce.shell [206.6] because it is exposed to package 'id.co.bippo.commerce.dao' from bundle revisions id.co.bippo.commerce.core [199.11] and id.co.bippo.commerce.core [199.12] via two dependency chains. Chain 1: id.co.bippo.commerce.shell [206.6] import : (&(osgi.wiring. package =id.co.bippo.commerce.dao)(version>=4.0.0)(!(version>=5.0.0))) | export: osgi.wiring. package =id.co.bippo.commerce.dao id.co.bippo.commerce.core [199.11] Chain 2: id.co.bippo.commerce.shell [206.6] import : (&(osgi.wiring. package =id.co.bippo.commerce.core)(version>=4.0.0)(!(version>=5.0.0))) | export: osgi.wiring. package =id.co.bippo.commerce.core; uses:=id.co.bippo.commerce.dao export: osgi.wiring. package =id.co.bippo.commerce.dao id.co.bippo.commerce.core [199.12] which seems to say that bundle 199 was not cleanly unloaded before updated? (so there are two versions 199.11 and 199.12)? Doing a: refresh 199 makes bundle 206 startable properly.
          Hide
          ceefour Hendy Irawan added a comment - - edited

          I'm confused... First I thought it was due to Aries JPA, but now my blueprint is practically empty:

          <?xml version="1.0" encoding="UTF-8"?>
          <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          	xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0"
          	xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
          	xmlns:jpa="http://aries.apache.org/xmlns/jpa/v1.0.0" xmlns:tx="http://aries.apache.org/xmlns/transactions/v1.0.0"
          	xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0
                  http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0
                  http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0 http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0
                  http://aries.apache.org/xmlns/jpa/v1.0.0 http://aries.apache.org/xmlns/jpa/v1.0.0
                  http://aries.apache.org/xmlns/transactions/v1.0.0 http://aries.apache.org/xmlns/transactions/v1.0.0">
          
          </blueprint>
          

          and I still got the Stopping status.

          I tried to do some grep and it seems there are other bundles in a half-state:

          
          karaf@root> list -s -t 0 | grep ing
          [   3] [Active     ] [            ] [    8] org.ops4j.pax.logging.pax-logging-service (1.6.9)
          [   4] [Active     ] [            ] [    8] org.ops4j.pax.logging.pax-logging-api (1.6.9)
          [  12] [Active     ] [Created     ] [   28] org.apache.karaf.deployer.spring (2.3.0.SNAPSHOT)
          [  50] [Starting   ] [            ] [   80] org.eclipse.emf.common (2.8.0.v20120606-0717)
          [  51] [Starting   ] [            ] [   80] org.eclipse.emf.ecore.xmi (2.8.0.v20120606-0717)
          [  52] [Starting   ] [            ] [   80] org.eclipse.emf.ecore (2.8.0.v20120606-0717)
          [ 190] [Active     ] [            ] [   50] wrap_mvn_com.google.code.morphia_morphia-logging-slf4j_0.99 (0)
          [ 199] [Stopping   ] [            ] [  100] id.co.bippo.commerce.core (4.0.1.SNAPSHOT)
          

          Maybe bundles 50,51,52 are the culprits?

          Update: No they aren't. Even when these 50,51,52 bundles are stopped (but resolved), I still get the Stopping bug:

          karaf@root> list -s -t 0 | grep ing
          [   3] [Active     ] [            ] [    8] org.ops4j.pax.logging.pax-logging-service (1.6.9)
          [   4] [Active     ] [            ] [    8] org.ops4j.pax.logging.pax-logging-api (1.6.9)
          [  12] [Active     ] [Created     ] [   28] org.apache.karaf.deployer.spring (2.3.0.SNAPSHOT)
          [ 190] [Active     ] [            ] [   50] wrap_mvn_com.google.code.morphia_morphia-logging-slf4j_0.99 (0)
          [ 199] [Stopping   ] [            ] [  100] id.co.bippo.commerce.core (4.0.1.SNAPSHOT)
          
          Show
          ceefour Hendy Irawan added a comment - - edited I'm confused... First I thought it was due to Aries JPA, but now my blueprint is practically empty: <?xml version= "1.0" encoding= "UTF-8" ?> <blueprint xmlns= "http: //www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance" xmlns:cm= "http: //aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0" xmlns:ext= "http: //aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0" xmlns:jpa= "http: //aries.apache.org/xmlns/jpa/v1.0.0" xmlns:tx= "http://aries.apache.org/xmlns/transactions/v1.0.0" xsi:schemaLocation="http: //www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0 http: //aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http: //aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0 http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0 http: //aries.apache.org/xmlns/jpa/v1.0.0 http://aries.apache.org/xmlns/jpa/v1.0.0 http: //aries.apache.org/xmlns/transactions/v1.0.0 http://aries.apache.org/xmlns/transactions/v1.0.0"> </blueprint> and I still got the Stopping status. I tried to do some grep and it seems there are other bundles in a half-state: karaf@root> list -s -t 0 | grep ing [ 3] [Active ] [ ] [ 8] org.ops4j.pax.logging.pax-logging-service (1.6.9) [ 4] [Active ] [ ] [ 8] org.ops4j.pax.logging.pax-logging-api (1.6.9) [ 12] [Active ] [Created ] [ 28] org.apache.karaf.deployer.spring (2.3.0.SNAPSHOT) [ 50] [Starting ] [ ] [ 80] org.eclipse.emf.common (2.8.0.v20120606-0717) [ 51] [Starting ] [ ] [ 80] org.eclipse.emf.ecore.xmi (2.8.0.v20120606-0717) [ 52] [Starting ] [ ] [ 80] org.eclipse.emf.ecore (2.8.0.v20120606-0717) [ 190] [Active ] [ ] [ 50] wrap_mvn_com.google.code.morphia_morphia-logging-slf4j_0.99 (0) [ 199] [Stopping ] [ ] [ 100] id.co.bippo.commerce.core (4.0.1.SNAPSHOT) Maybe bundles 50,51,52 are the culprits? Update: No they aren't. Even when these 50,51,52 bundles are stopped (but resolved), I still get the Stopping bug: karaf@root> list -s -t 0 | grep ing [ 3] [Active ] [ ] [ 8] org.ops4j.pax.logging.pax-logging-service (1.6.9) [ 4] [Active ] [ ] [ 8] org.ops4j.pax.logging.pax-logging-api (1.6.9) [ 12] [Active ] [Created ] [ 28] org.apache.karaf.deployer.spring (2.3.0.SNAPSHOT) [ 190] [Active ] [ ] [ 50] wrap_mvn_com.google.code.morphia_morphia-logging-slf4j_0.99 (0) [ 199] [Stopping ] [ ] [ 100] id.co.bippo.commerce.core (4.0.1.SNAPSHOT)
          Hide
          ceefour Hendy Irawan added a comment -

          This is what Blueprint says when bundle (with empty Blueprint XML) is currently Resolved, then I start it (successfully):

          2012-08-15 17:09:30,703 | DEBUG | lixDispatchQueue | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent RESOLVED
          2012-08-15 17:09:30,703 | DEBUG | l Console Thread | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent REGISTERED
          2012-08-15 17:09:30,705 | DEBUG | lixDispatchQueue | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent STARTED
          2012-08-15 17:09:30,708 | DEBUG | rint Extender: 2 | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent REGISTERED
          

          Then, after doing "update 199; refresh 199", and then stuck in "Stopping", here was what Blueprint logged:

          2012-08-15 17:09:44,769 | DEBUG | l Console Thread | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent UNREGISTERING
          2012-08-15 17:09:44,772 | DEBUG | l Console Thread | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent UNREGISTERING
          2012-08-15 17:09:44,797 | DEBUG | lixDispatchQueue | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent STOPPED
          2012-08-15 17:09:44,797 | DEBUG | lixDispatchQueue | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent UNRESOLVED
          2012-08-15 17:09:44,806 | DEBUG | lixDispatchQueue | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent UPDATED
          2012-08-15 17:09:44,820 | DEBUG | lixDispatchQueue | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent RESOLVED
          2012-08-15 17:09:44,820 | DEBUG | l Console Thread | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent REGISTERED
          2012-08-15 17:09:44,823 | DEBUG | lixDispatchQueue | core                             | ?                                   ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent STARTED
          
          Show
          ceefour Hendy Irawan added a comment - This is what Blueprint says when bundle (with empty Blueprint XML) is currently Resolved, then I start it (successfully): 2012-08-15 17:09:30,703 | DEBUG | lixDispatchQueue | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent RESOLVED 2012-08-15 17:09:30,703 | DEBUG | l Console Thread | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent REGISTERED 2012-08-15 17:09:30,705 | DEBUG | lixDispatchQueue | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent STARTED 2012-08-15 17:09:30,708 | DEBUG | rint Extender: 2 | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent REGISTERED Then, after doing "update 199; refresh 199", and then stuck in "Stopping", here was what Blueprint logged: 2012-08-15 17:09:44,769 | DEBUG | l Console Thread | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent UNREGISTERING 2012-08-15 17:09:44,772 | DEBUG | l Console Thread | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent UNREGISTERING 2012-08-15 17:09:44,797 | DEBUG | lixDispatchQueue | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent STOPPED 2012-08-15 17:09:44,797 | DEBUG | lixDispatchQueue | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent UNRESOLVED 2012-08-15 17:09:44,806 | DEBUG | lixDispatchQueue | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent UPDATED 2012-08-15 17:09:44,820 | DEBUG | lixDispatchQueue | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent RESOLVED 2012-08-15 17:09:44,820 | DEBUG | l Console Thread | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | ServiceEvent REGISTERED 2012-08-15 17:09:44,823 | DEBUG | lixDispatchQueue | core | ? ? | 199 - id.co.bippo.commerce.core - 4.0.1.SNAPSHOT | BundleEvent STARTED
          Hide
          ceefour Hendy Irawan added a comment -

          Deleting the Blueprint XML file entirely makes the update/refresh/dev:watch works fine. But of course now my bundle doesn't do anything.

          Curiously, with an empty Blueprint XML, my bundle still doesn't do anything, but it can make Blueprint stuck.

          But at least now I know it's not because some Java code running (static initializer etc.), the bundle doesn't any OSGi activators. It seems a Blueprint issue.

          Show
          ceefour Hendy Irawan added a comment - Deleting the Blueprint XML file entirely makes the update/refresh/dev:watch works fine. But of course now my bundle doesn't do anything. Curiously, with an empty Blueprint XML, my bundle still doesn't do anything, but it can make Blueprint stuck. But at least now I know it's not because some Java code running (static initializer etc.), the bundle doesn't any OSGi activators. It seems a Blueprint issue.
          Hide
          ceefour Hendy Irawan added a comment -

          Curious again, why it says "Stopping", although the logs mentioned that the last BundlEvent is "STARTED"

          I also tried with the following Blueprint XML (without any namespace handlers):

          <?xml version="1.0" encoding="UTF-8"?>
          <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
          </blueprint>
          

          and it's still "Stopping".

          Show
          ceefour Hendy Irawan added a comment - Curious again, why it says "Stopping", although the logs mentioned that the last BundlEvent is "STARTED" I also tried with the following Blueprint XML (without any namespace handlers): <?xml version= "1.0" encoding= "UTF-8" ?> <blueprint xmlns= "http: //www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance" > </blueprint> and it's still "Stopping".
          Hide
          ceefour Hendy Irawan added a comment - - edited

          It seems there is a workaround:

          1. stop <bundle>
          2. wait until it is actually stopped (status=Resolved)
          3. update <bundle>; refresh <bundle>
          4. start <bundle>

          In early tests, it seems this works.

          If so, then the problem seems to be because Blueprint/Karaf rushes too fast to update/start the bundle before it (the bundle's Blueprint context) was fully stopped.

          This little command chain is a workaround:

          stop 199; sleep 1000; update 199; refresh 199; start 199
          
          Show
          ceefour Hendy Irawan added a comment - - edited It seems there is a workaround: 1. stop <bundle> 2. wait until it is actually stopped (status=Resolved) 3. update <bundle>; refresh <bundle> 4. start <bundle> In early tests, it seems this works. If so, then the problem seems to be because Blueprint/Karaf rushes too fast to update/start the bundle before it (the bundle's Blueprint context) was fully stopped. This little command chain is a workaround: stop 199; sleep 1000; update 199; refresh 199; start 199
          Hide
          sgo Sebastian Gozin added a comment -

          I can easily reproduce this issue with the following example application. https://github.com/sgo/osgisandbox (commit:68961c118478b2bdf5113e2dc44e1b07a62e8180)

          Simply add <blueprint default-availability="optional"/> in all 3 blueprint descriptor files (rest.xml, interactors.xml, inmemorystore.xml) and install the jars. gradle uploadArchives will copy them all to $rootProject/karaf/bundle. You may need to touch any one of the jar files which for some reason triggers a redeploy of all jars and not just the one touched. (there's also a dependency on cxf, jackson, guava and groovy)

          End result all blueprint bundles are stuck in a deadlock.

          Show
          sgo Sebastian Gozin added a comment - I can easily reproduce this issue with the following example application. https://github.com/sgo/osgisandbox (commit:68961c118478b2bdf5113e2dc44e1b07a62e8180) Simply add <blueprint default-availability="optional"/> in all 3 blueprint descriptor files (rest.xml, interactors.xml, inmemorystore.xml) and install the jars. gradle uploadArchives will copy them all to $rootProject/karaf/bundle. You may need to touch any one of the jar files which for some reason triggers a redeploy of all jars and not just the one touched. (there's also a dependency on cxf, jackson, guava and groovy) End result all blueprint bundles are stuck in a deadlock.
          Hide
          neduz Tom Mercelis added a comment -

          I also have this problem where Blueprint bundles get stuck in "Stopping" state. I've read that JConsole can detect deadlocks, but no deadlocks can be detected by JConsole in my running Karaf with a "Stopping" bundle.

          Recently I investigated a Karaf instance that got stuck at startup. While a few of the deploy/*.xml files have been created according to osgi:list output, some are still in "Resolved" state and one is in "Stopping". (I don't know why bundles enter "stopping" during a startup ... possibly the file in the deploy folder changed between the last and this run of Karaf...)

          But, as it was happening I tried to figure out what was happening with all the threads. I found a few threads that seem to locking/releasing the same object really fast (after minutes of running the counters "total blocked", "total waited" were in the millions:

          (PS: these thread-stacktraces were copied over a period of several minutes, that is why the numbers start in the 40.000.000 and vary by a few tens of millions.)
          You'll also get the impression threads are in state WAITING, but that's just to should that they all lock on the same object... in reality the state varies very quickly for each of these threads, I just re-read the information until it showed the object it locked on.

          Name: CM Configuration Updater (ManagedServiceFactory Update: factoryPid=[org.apache.activemq.server])
          State: WAITING on [Ljava.lang.Object;@fe814a
          Total blocked: 40.680.837  Total waited: 40.680.784
          
          Stack trace: 
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944)
          org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:219)
          org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1539)
          org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1439)
          org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
          org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
          java.lang.ClassLoader.loadClass(ClassLoader.java:356)
          java.lang.Class.forName0(Native Method)
          java.lang.Class.forName(Class.java:264)
          com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:67)
          com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:110)
          com.sun.beans.finder.InstanceFinder.instantiate(InstanceFinder.java:93)
          com.sun.beans.finder.InstanceFinder.find(InstanceFinder.java:66)
          java.beans.Introspector.findExplicitBeanInfo(Introspector.java:436)
          java.beans.Introspector.<init>(Introspector.java:386)
          java.beans.Introspector.getBeanInfo(Introspector.java:161)
          org.springframework.beans.CachedIntrospectionResults.<init>(CachedIntrospectionResults.java:217)
          org.springframework.beans.CachedIntrospectionResults.forClass(CachedIntrospectionResults.java:149)
          org.springframework.beans.BeanWrapperImpl.getCachedIntrospectionResults(BeanWrapperImpl.java:324)
          org.springframework.beans.BeanWrapperImpl.getPropertyDescriptorInternal(BeanWrapperImpl.java:354)
          org.springframework.beans.BeanWrapperImpl.isWritableProperty(BeanWrapperImpl.java:430)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1362)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
          org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:271)
          org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:126)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
          org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:271)
          org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:121)
          org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:354)
          org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:154)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
          org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:271)
          org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:126)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
          org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
          org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
          org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
             - locked java.util.concurrent.ConcurrentHashMap@1a60b3e
          org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
          org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
          org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:607)
          org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925)
          org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:472)
             - locked java.lang.Object@2c21a0
          org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:91)
             - locked org.apache.activemq.osgi.ActiveMQServiceFactory@18364b6
          org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:88)
          org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1593)
          org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1536)
          org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
          java.lang.Thread.run(Thread.java:722)
          

          And another thread:

          Name: fileinstall-C:\apache-karaf-2.3.1/deploy
          State: WAITING on [Ljava.lang.Object;@fe814a
          Total blocked: 70.092.531  Total waited: 70.092.549
          
          Stack trace: 
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871)
          org.apache.felix.framework.Felix.startBundle(Felix.java:1744)
          org.apache.felix.framework.BundleImpl.start(BundleImpl.java:944)
          org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1247)
          org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1219)
          org.apache.felix.fileinstall.internal.DirectoryWatcher.startAllBundles(DirectoryWatcher.java:1208)
          org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:503)
          org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:291)
          
          Name: Executor: 1
          State: WAITING on [Ljava.lang.Object;@fe814a
          Total blocked: 55.181.647  Total waited: 55.181.633
          
          Stack trace: 
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944)
          org.apache.felix.framework.Felix.resolveBundles(Felix.java:3736)
          org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1622)
          org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:303)
          org.ops4j.pax.web.extender.war.internal.WebAppHttpContext.getResource(WebAppHttpContext.java:129)
          org.ops4j.pax.web.service.jetty.internal.HttpServiceContext$SContext$1.run(HttpServiceContext.java:286)
          org.ops4j.pax.web.service.jetty.internal.HttpServiceContext$SContext$1.run(HttpServiceContext.java:282)
          java.security.AccessController.doPrivileged(Native Method)
          org.ops4j.pax.web.service.jetty.internal.HttpServiceContext$SContext.getResource(HttpServiceContext.java:280)
          org.apache.jasper.compiler.JspConfig.processWebDotXml(JspConfig.java:126)
          org.apache.jasper.compiler.JspConfig.init(JspConfig.java:312)
          org.apache.jasper.compiler.JspConfig.<init>(JspConfig.java:118)
          org.apache.jasper.EmbeddedServletOptions.<init>(EmbeddedServletOptions.java:717)
          org.apache.jasper.servlet.JspServlet.init(JspServlet.java:149)
          org.ops4j.pax.web.jsp.JspServletWrapper$1.call(JspServletWrapper.java:89)
          org.ops4j.pax.web.jsp.JspServletWrapper$1.call(JspServletWrapper.java:84)
          org.ops4j.pax.swissbox.core.ContextClassLoaderUtils.doWithClassLoader(ContextClassLoaderUtils.java:60)
          org.ops4j.pax.web.jsp.JspServletWrapper.init(JspServletWrapper.java:81)
          org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:477)
          org.eclipse.jetty.servlet.ServletHolder.doStart(ServletHolder.java:293)
          org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
             - locked java.lang.Object@41a6a3
          org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:739)
          org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:254)
          org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:683)
          org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doStart(HttpServiceContext.java:100)
          org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
             - locked java.lang.Object@234499
          org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$1.start(JettyServerImpl.java:165)
          org.ops4j.pax.web.service.internal.HttpServiceStarted.end(HttpServiceStarted.java:758)
          org.ops4j.pax.web.service.internal.HttpServiceProxy.end(HttpServiceProxy.java:281)
          org.ops4j.pax.web.extender.war.internal.RegisterWebAppVisitorWC.end(RegisterWebAppVisitorWC.java:348)
          org.ops4j.pax.web.extender.war.internal.model.WebApp.accept(WebApp.java:629)
          org.ops4j.pax.web.extender.war.internal.WebAppPublisher$HttpServiceListener.register(WebAppPublisher.java:170)
          org.ops4j.pax.web.extender.war.internal.WebAppPublisher$HttpServiceListener.serviceChanged(WebAppPublisher.java:155)
             - locked org.ops4j.pax.web.extender.war.internal.WebAppPublisher$HttpServiceListener@d0a62f
          org.ops4j.pax.web.extender.war.internal.WebAppPublisher$HttpServiceListener.serviceChanged(WebAppPublisher.java:119)
          org.ops4j.pax.swissbox.tracker.ReplaceableService.setService(ReplaceableService.java:114)
             - locked org.ops4j.pax.swissbox.tracker.ReplaceableService@1ad4bf6
          org.ops4j.pax.swissbox.tracker.ReplaceableService.access$100(ReplaceableService.java:28)
          org.ops4j.pax.swissbox.tracker.ReplaceableService$CollectionListener.serviceAdded(ReplaceableService.java:183)
          org.ops4j.pax.swissbox.tracker.ServiceCollection$Tracker.addingService(ServiceCollection.java:181)
          org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932)
          org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:1)
          org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)
          org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183)
          org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:317)
          org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261)
          org.ops4j.pax.swissbox.tracker.ServiceCollection.onStart(ServiceCollection.java:139)
          org.ops4j.pax.swissbox.lifecycle.AbstractLifecycle$Stopped.start(AbstractLifecycle.java:121)
          org.ops4j.pax.swissbox.lifecycle.AbstractLifecycle.start(AbstractLifecycle.java:49)
          org.ops4j.pax.swissbox.tracker.ReplaceableService.onStart(ReplaceableService.java:146)
          org.ops4j.pax.swissbox.lifecycle.AbstractLifecycle$Stopped.start(AbstractLifecycle.java:121)
          org.ops4j.pax.swissbox.lifecycle.AbstractLifecycle.start(AbstractLifecycle.java:49)
          org.ops4j.pax.web.extender.war.internal.WebAppPublisher.publish(WebAppPublisher.java:81)
          org.ops4j.pax.web.extender.war.internal.WebXmlObserver.deploy(WebXmlObserver.java:204)
          org.ops4j.pax.web.extender.war.internal.WebXmlObserver.addingEntries(WebXmlObserver.java:162)
          org.ops4j.pax.swissbox.extender.BundleWatcher$3.run(BundleWatcher.java:224)
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          java.lang.Thread.run(Thread.java:722)
          
          Name: Blueprint Extender: 1
          State: BLOCKED on [Ljava.lang.Object;@fe814a owned by: Executor: 1
          Total blocked: 58.352.836  Total waited: 58.352.816
          
          Stack trace: 
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944)
          org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:219)
          org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1539)
          org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1439)
          org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
          org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
          java.lang.ClassLoader.loadClass(ClassLoader.java:410)
             - locked org.apache.aries.proxy.impl.interfaces.ProxyClassLoader@162ff82
          java.lang.ClassLoader.loadClass(ClassLoader.java:356)
          java.lang.ClassLoader.defineClass1(Native Method)
          java.lang.ClassLoader.defineClass(ClassLoader.java:791)
          org.apache.aries.proxy.impl.interfaces.ProxyClassLoader.createProxyClass(ProxyClassLoader.java:157)
          org.apache.aries.proxy.impl.interfaces.InterfaceProxyGenerator.getProxyInstance(InterfaceProxyGenerator.java:95)
          org.apache.aries.proxy.impl.AsmProxyManager.createNewProxy(AsmProxyManager.java:72)
          org.apache.aries.proxy.impl.AbstractProxyManager.createDelegatingInterceptingProxy(AbstractProxyManager.java:75)
          org.apache.aries.proxy.impl.AbstractProxyManager.createDelegatingProxy(AbstractProxyManager.java:40)
          org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.createProxy(AbstractServiceReferenceRecipe.java:306)
          org.apache.aries.blueprint.container.ReferenceRecipe.internalCreate(ReferenceRecipe.java:99)
          org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)
          org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:245)
          org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:183)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:668)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:370)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261)
             - locked java.util.concurrent.atomic.AtomicBoolean@19dbd6a
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106)
          org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          java.lang.Thread.run(Thread.java:722)
          
          Name: Blueprint Extender: 2
          State: BLOCKED on [Ljava.lang.Object;@fe814a owned by: Executor: 1
          Total blocked: 87.317.768  Total waited: 87.317.783
          
          Stack trace: 
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871)
          org.apache.felix.framework.Felix.registerService(Felix.java:3205)
          org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.registerService(BlueprintContainerImpl.java:439)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:379)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261)
             - locked java.util.concurrent.atomic.AtomicBoolean@43cb07
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106)
          org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          java.lang.Thread.run(Thread.java:722)
          
          Name: Blueprint Extender: 3
          State: WAITING on [Ljava.lang.Object;@fe814a
          Total blocked: 63.342.290  Total waited: 63.342.252
          
          Stack trace: 
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944)
          org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:219)
          org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1539)
          org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1439)
          org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
          org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
          java.lang.ClassLoader.loadClass(ClassLoader.java:410)
             - locked org.apache.aries.proxy.impl.interfaces.ProxyClassLoader@1a587da
          java.lang.ClassLoader.loadClass(ClassLoader.java:356)
          java.lang.ClassLoader.defineClass1(Native Method)
          java.lang.ClassLoader.defineClass(ClassLoader.java:791)
          org.apache.aries.proxy.impl.interfaces.ProxyClassLoader.createProxyClass(ProxyClassLoader.java:157)
          org.apache.aries.proxy.impl.interfaces.InterfaceProxyGenerator.getProxyInstance(InterfaceProxyGenerator.java:95)
          org.apache.aries.proxy.impl.AsmProxyManager.createNewProxy(AsmProxyManager.java:72)
          org.apache.aries.proxy.impl.AbstractProxyManager.createDelegatingInterceptingProxy(AbstractProxyManager.java:75)
          org.apache.aries.proxy.impl.AbstractProxyManager.createDelegatingProxy(AbstractProxyManager.java:40)
          org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.createProxy(AbstractServiceReferenceRecipe.java:306)
          org.apache.aries.blueprint.container.ReferenceRecipe.internalCreate(ReferenceRecipe.java:99)
          org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)
          org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:245)
          org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:183)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:668)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:370)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261)
             - locked java.util.concurrent.atomic.AtomicBoolean@17352d7
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106)
          org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          java.lang.Thread.run(Thread.java:722)
          
          Show
          neduz Tom Mercelis added a comment - I also have this problem where Blueprint bundles get stuck in "Stopping" state. I've read that JConsole can detect deadlocks, but no deadlocks can be detected by JConsole in my running Karaf with a "Stopping" bundle. Recently I investigated a Karaf instance that got stuck at startup. While a few of the deploy/*.xml files have been created according to osgi:list output, some are still in "Resolved" state and one is in "Stopping". (I don't know why bundles enter "stopping" during a startup ... possibly the file in the deploy folder changed between the last and this run of Karaf...) But, as it was happening I tried to figure out what was happening with all the threads. I found a few threads that seem to locking/releasing the same object really fast (after minutes of running the counters "total blocked", "total waited" were in the millions: (PS: these thread-stacktraces were copied over a period of several minutes, that is why the numbers start in the 40.000.000 and vary by a few tens of millions.) You'll also get the impression threads are in state WAITING, but that's just to should that they all lock on the same object... in reality the state varies very quickly for each of these threads, I just re-read the information until it showed the object it locked on. Name: CM Configuration Updater (ManagedServiceFactory Update: factoryPid=[org.apache.activemq.server]) State: WAITING on [Ljava.lang.Object;@fe814a Total blocked: 40.680.837 Total waited: 40.680.784 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944) org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:219) org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1539) org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1439) org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72) org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843) java.lang.ClassLoader.loadClass(ClassLoader.java:356) java.lang.Class.forName0(Native Method) java.lang.Class.forName(Class.java:264) com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:67) com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:110) com.sun.beans.finder.InstanceFinder.instantiate(InstanceFinder.java:93) com.sun.beans.finder.InstanceFinder.find(InstanceFinder.java:66) java.beans.Introspector.findExplicitBeanInfo(Introspector.java:436) java.beans.Introspector.<init>(Introspector.java:386) java.beans.Introspector.getBeanInfo(Introspector.java:161) org.springframework.beans.CachedIntrospectionResults.<init>(CachedIntrospectionResults.java:217) org.springframework.beans.CachedIntrospectionResults.forClass(CachedIntrospectionResults.java:149) org.springframework.beans.BeanWrapperImpl.getCachedIntrospectionResults(BeanWrapperImpl.java:324) org.springframework.beans.BeanWrapperImpl.getPropertyDescriptorInternal(BeanWrapperImpl.java:354) org.springframework.beans.BeanWrapperImpl.isWritableProperty(BeanWrapperImpl.java:430) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1362) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:271) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:126) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:271) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:121) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:354) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:154) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:271) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:126) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) - locked java.util.concurrent.ConcurrentHashMap@1a60b3e org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:607) org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925) org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:472) - locked java.lang.Object@2c21a0 org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:91) - locked org.apache.activemq.osgi.ActiveMQServiceFactory@18364b6 org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:88) org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1593) org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1536) org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103) java.lang.Thread.run(Thread.java:722) And another thread: Name: fileinstall-C:\apache-karaf-2.3.1/deploy State: WAITING on [Ljava.lang.Object;@fe814a Total blocked: 70.092.531 Total waited: 70.092.549 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871) org.apache.felix.framework.Felix.startBundle(Felix.java:1744) org.apache.felix.framework.BundleImpl.start(BundleImpl.java:944) org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1247) org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1219) org.apache.felix.fileinstall.internal.DirectoryWatcher.startAllBundles(DirectoryWatcher.java:1208) org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:503) org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:291) Name: Executor: 1 State: WAITING on [Ljava.lang.Object;@fe814a Total blocked: 55.181.647 Total waited: 55.181.633 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944) org.apache.felix.framework.Felix.resolveBundles(Felix.java:3736) org.apache.felix.framework.Felix.findBundleEntries(Felix.java:1622) org.apache.felix.framework.BundleImpl.findEntries(BundleImpl.java:303) org.ops4j.pax.web.extender.war.internal.WebAppHttpContext.getResource(WebAppHttpContext.java:129) org.ops4j.pax.web.service.jetty.internal.HttpServiceContext$SContext$1.run(HttpServiceContext.java:286) org.ops4j.pax.web.service.jetty.internal.HttpServiceContext$SContext$1.run(HttpServiceContext.java:282) java.security.AccessController.doPrivileged(Native Method) org.ops4j.pax.web.service.jetty.internal.HttpServiceContext$SContext.getResource(HttpServiceContext.java:280) org.apache.jasper.compiler.JspConfig.processWebDotXml(JspConfig.java:126) org.apache.jasper.compiler.JspConfig.init(JspConfig.java:312) org.apache.jasper.compiler.JspConfig.<init>(JspConfig.java:118) org.apache.jasper.EmbeddedServletOptions.<init>(EmbeddedServletOptions.java:717) org.apache.jasper.servlet.JspServlet.init(JspServlet.java:149) org.ops4j.pax.web.jsp.JspServletWrapper$1.call(JspServletWrapper.java:89) org.ops4j.pax.web.jsp.JspServletWrapper$1.call(JspServletWrapper.java:84) org.ops4j.pax.swissbox.core.ContextClassLoaderUtils.doWithClassLoader(ContextClassLoaderUtils.java:60) org.ops4j.pax.web.jsp.JspServletWrapper.init(JspServletWrapper.java:81) org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:477) org.eclipse.jetty.servlet.ServletHolder.doStart(ServletHolder.java:293) org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) - locked java.lang.Object@41a6a3 org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:739) org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:254) org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:683) org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doStart(HttpServiceContext.java:100) org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) - locked java.lang.Object@234499 org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$1.start(JettyServerImpl.java:165) org.ops4j.pax.web.service.internal.HttpServiceStarted.end(HttpServiceStarted.java:758) org.ops4j.pax.web.service.internal.HttpServiceProxy.end(HttpServiceProxy.java:281) org.ops4j.pax.web.extender.war.internal.RegisterWebAppVisitorWC.end(RegisterWebAppVisitorWC.java:348) org.ops4j.pax.web.extender.war.internal.model.WebApp.accept(WebApp.java:629) org.ops4j.pax.web.extender.war.internal.WebAppPublisher$HttpServiceListener.register(WebAppPublisher.java:170) org.ops4j.pax.web.extender.war.internal.WebAppPublisher$HttpServiceListener.serviceChanged(WebAppPublisher.java:155) - locked org.ops4j.pax.web.extender.war.internal.WebAppPublisher$HttpServiceListener@d0a62f org.ops4j.pax.web.extender.war.internal.WebAppPublisher$HttpServiceListener.serviceChanged(WebAppPublisher.java:119) org.ops4j.pax.swissbox.tracker.ReplaceableService.setService(ReplaceableService.java:114) - locked org.ops4j.pax.swissbox.tracker.ReplaceableService@1ad4bf6 org.ops4j.pax.swissbox.tracker.ReplaceableService.access$100(ReplaceableService.java:28) org.ops4j.pax.swissbox.tracker.ReplaceableService$CollectionListener.serviceAdded(ReplaceableService.java:183) org.ops4j.pax.swissbox.tracker.ServiceCollection$Tracker.addingService(ServiceCollection.java:181) org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932) org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:1) org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256) org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183) org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:317) org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261) org.ops4j.pax.swissbox.tracker.ServiceCollection.onStart(ServiceCollection.java:139) org.ops4j.pax.swissbox.lifecycle.AbstractLifecycle$Stopped.start(AbstractLifecycle.java:121) org.ops4j.pax.swissbox.lifecycle.AbstractLifecycle.start(AbstractLifecycle.java:49) org.ops4j.pax.swissbox.tracker.ReplaceableService.onStart(ReplaceableService.java:146) org.ops4j.pax.swissbox.lifecycle.AbstractLifecycle$Stopped.start(AbstractLifecycle.java:121) org.ops4j.pax.swissbox.lifecycle.AbstractLifecycle.start(AbstractLifecycle.java:49) org.ops4j.pax.web.extender.war.internal.WebAppPublisher.publish(WebAppPublisher.java:81) org.ops4j.pax.web.extender.war.internal.WebXmlObserver.deploy(WebXmlObserver.java:204) org.ops4j.pax.web.extender.war.internal.WebXmlObserver.addingEntries(WebXmlObserver.java:162) org.ops4j.pax.swissbox.extender.BundleWatcher$3.run(BundleWatcher.java:224) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) Name: Blueprint Extender: 1 State: BLOCKED on [Ljava.lang.Object;@fe814a owned by: Executor: 1 Total blocked: 58.352.836 Total waited: 58.352.816 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944) org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:219) org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1539) org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1439) org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72) org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843) java.lang.ClassLoader.loadClass(ClassLoader.java:410) - locked org.apache.aries.proxy.impl.interfaces.ProxyClassLoader@162ff82 java.lang.ClassLoader.loadClass(ClassLoader.java:356) java.lang.ClassLoader.defineClass1(Native Method) java.lang.ClassLoader.defineClass(ClassLoader.java:791) org.apache.aries.proxy.impl.interfaces.ProxyClassLoader.createProxyClass(ProxyClassLoader.java:157) org.apache.aries.proxy.impl.interfaces.InterfaceProxyGenerator.getProxyInstance(InterfaceProxyGenerator.java:95) org.apache.aries.proxy.impl.AsmProxyManager.createNewProxy(AsmProxyManager.java:72) org.apache.aries.proxy.impl.AbstractProxyManager.createDelegatingInterceptingProxy(AbstractProxyManager.java:75) org.apache.aries.proxy.impl.AbstractProxyManager.createDelegatingProxy(AbstractProxyManager.java:40) org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.createProxy(AbstractServiceReferenceRecipe.java:306) org.apache.aries.blueprint.container.ReferenceRecipe.internalCreate(ReferenceRecipe.java:99) org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88) org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:245) org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:183) org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:668) org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:370) org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261) - locked java.util.concurrent.atomic.AtomicBoolean@19dbd6a java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106) org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) Name: Blueprint Extender: 2 State: BLOCKED on [Ljava.lang.Object;@fe814a owned by: Executor: 1 Total blocked: 87.317.768 Total waited: 87.317.783 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871) org.apache.felix.framework.Felix.registerService(Felix.java:3205) org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346) org.apache.aries.blueprint.container.BlueprintContainerImpl.registerService(BlueprintContainerImpl.java:439) org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:379) org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261) - locked java.util.concurrent.atomic.AtomicBoolean@43cb07 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106) org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) Name: Blueprint Extender: 3 State: WAITING on [Ljava.lang.Object;@fe814a Total blocked: 63.342.290 Total waited: 63.342.252 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944) org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:219) org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1539) org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1439) org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72) org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843) java.lang.ClassLoader.loadClass(ClassLoader.java:410) - locked org.apache.aries.proxy.impl.interfaces.ProxyClassLoader@1a587da java.lang.ClassLoader.loadClass(ClassLoader.java:356) java.lang.ClassLoader.defineClass1(Native Method) java.lang.ClassLoader.defineClass(ClassLoader.java:791) org.apache.aries.proxy.impl.interfaces.ProxyClassLoader.createProxyClass(ProxyClassLoader.java:157) org.apache.aries.proxy.impl.interfaces.InterfaceProxyGenerator.getProxyInstance(InterfaceProxyGenerator.java:95) org.apache.aries.proxy.impl.AsmProxyManager.createNewProxy(AsmProxyManager.java:72) org.apache.aries.proxy.impl.AbstractProxyManager.createDelegatingInterceptingProxy(AbstractProxyManager.java:75) org.apache.aries.proxy.impl.AbstractProxyManager.createDelegatingProxy(AbstractProxyManager.java:40) org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.createProxy(AbstractServiceReferenceRecipe.java:306) org.apache.aries.blueprint.container.ReferenceRecipe.internalCreate(ReferenceRecipe.java:99) org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88) org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:245) org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:183) org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:668) org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:370) org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261) - locked java.util.concurrent.atomic.AtomicBoolean@17352d7 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106) org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722)
          Hide
          neduz Tom Mercelis added a comment -

          And I have another situation, this time I created the situation by starting a watch on a bundle, then redeployed the watched bundle, and it got stuck in "stopping".

          One time I saw a very similar behavior as in my previous comment: the very rapidly passing of the lock on 1 object, this time it ran between the Threads "Extender 1", "fileinstall-C:\apache-karaf-2.3.1/deploy" and a thread created by my own code which was trying to unregister a service and the "Pax Web Runtime worker".

          But I also ran into a condition where the lock wasn't going around very fast. This situation was also initiated while dev:watch was updating a bundle (a bundle on which a few other bundles depend). I tried stopping another bundle manually, but that locked the osgi:stop command. Below some of the threads that seemed interesting to me:

          Name: Karaf Shell Console Thread
          State: WAITING on [Ljava.lang.Object;@1226691
          Total blocked: 3  Total waited: 153
          
          Stack trace: 
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871)
          org.apache.felix.framework.Felix.stopBundle(Felix.java:2279)
          org.apache.felix.framework.BundleImpl.stop(BundleImpl.java:980)
          org.apache.felix.framework.BundleImpl.stop(BundleImpl.java:967)
          org.apache.karaf.shell.osgi.StopBundle.doExecute(StopBundle.java:37)
          org.apache.karaf.shell.osgi.BundlesCommand.doExecute(BundlesCommand.java:37)
          org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38)
          org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)
          org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78)
          org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:474)
          org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:400)
          org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
          org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)
          org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)
          org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:89)
          org.apache.karaf.shell.console.jline.Console.run(Console.java:174)
          java.lang.Thread.run(Thread.java:722)
          

          This blueprint thread is also waiting on that object, but notice it holds a lock on AtomicBoolean@ca2ddb

          Name: Blueprint Extender: 1
          State: WAITING on [Ljava.lang.Object;@1226691
          Total blocked: 1.667  Total waited: 1.781
          
          Stack trace: 
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944)
          org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:102)
          org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3819)
          org.apache.felix.framework.Felix.getBundleResource(Felix.java:1498)
          org.apache.felix.framework.BundleImpl.getResource(BundleImpl.java:628)
          org.apache.aries.blueprint.namespace.NamespaceHandlerRegistryImpl$NamespaceHandlerSetImpl.<init>(NamespaceHandlerRegistryImpl.java:507)
          org.apache.aries.blueprint.namespace.NamespaceHandlerRegistryImpl.getNamespaceHandlers(NamespaceHandlerRegistryImpl.java:229)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:288)
          org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261)
             - locked java.util.concurrent.atomic.AtomicBoolean@ca2ddb
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106)
          org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          java.lang.Thread.run(Thread.java:722)
          

          And this thread is blocked on that AtomicBoolean...

          Name: FelixFrameworkWiring
          State: BLOCKED on java.util.concurrent.atomic.AtomicBoolean@ca2ddb owned by: Blueprint Extender: 1
          Total blocked: 9  Total waited: 12
          
          Stack trace: 
          org.apache.aries.blueprint.container.BlueprintContainerImpl.destroy(BlueprintContainerImpl.java:842)
          org.apache.aries.blueprint.container.BlueprintExtender$3.run(BlueprintExtender.java:284)
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          java.util.concurrent.FutureTask.run(FutureTask.java:166)
          org.apache.aries.blueprint.container.BlueprintExtender.destroyContainer(BlueprintExtender.java:305)
          org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:206)
          org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)
          org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)
          org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)
          org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)
          org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)
          org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)
          org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)
          org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)
          org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4244)
          org.apache.felix.framework.Felix.stopBundle(Felix.java:2351)
          org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4629)
          org.apache.felix.framework.Felix.refreshPackages(Felix.java:3951)
          org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:172)
          java.lang.Thread.run(Thread.java:722)
          
          Show
          neduz Tom Mercelis added a comment - And I have another situation, this time I created the situation by starting a watch on a bundle, then redeployed the watched bundle, and it got stuck in "stopping". One time I saw a very similar behavior as in my previous comment: the very rapidly passing of the lock on 1 object, this time it ran between the Threads "Extender 1", "fileinstall-C:\apache-karaf-2.3.1/deploy" and a thread created by my own code which was trying to unregister a service and the "Pax Web Runtime worker". But I also ran into a condition where the lock wasn't going around very fast. This situation was also initiated while dev:watch was updating a bundle (a bundle on which a few other bundles depend). I tried stopping another bundle manually, but that locked the osgi:stop command. Below some of the threads that seemed interesting to me: Name: Karaf Shell Console Thread State: WAITING on [Ljava.lang.Object;@1226691 Total blocked: 3 Total waited: 153 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871) org.apache.felix.framework.Felix.stopBundle(Felix.java:2279) org.apache.felix.framework.BundleImpl.stop(BundleImpl.java:980) org.apache.felix.framework.BundleImpl.stop(BundleImpl.java:967) org.apache.karaf.shell.osgi.StopBundle.doExecute(StopBundle.java:37) org.apache.karaf.shell.osgi.BundlesCommand.doExecute(BundlesCommand.java:37) org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38) org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35) org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78) org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:474) org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:400) org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108) org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183) org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120) org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:89) org.apache.karaf.shell.console.jline.Console.run(Console.java:174) java.lang.Thread.run(Thread.java:722) This blueprint thread is also waiting on that object, but notice it holds a lock on AtomicBoolean@ca2ddb Name: Blueprint Extender: 1 State: WAITING on [Ljava.lang.Object;@1226691 Total blocked: 1.667 Total waited: 1.781 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944) org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:102) org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3819) org.apache.felix.framework.Felix.getBundleResource(Felix.java:1498) org.apache.felix.framework.BundleImpl.getResource(BundleImpl.java:628) org.apache.aries.blueprint.namespace.NamespaceHandlerRegistryImpl$NamespaceHandlerSetImpl.<init>(NamespaceHandlerRegistryImpl.java:507) org.apache.aries.blueprint.namespace.NamespaceHandlerRegistryImpl.getNamespaceHandlers(NamespaceHandlerRegistryImpl.java:229) org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:288) org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261) - locked java.util.concurrent.atomic.AtomicBoolean@ca2ddb java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106) org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) And this thread is blocked on that AtomicBoolean... Name: FelixFrameworkWiring State: BLOCKED on java.util.concurrent.atomic.AtomicBoolean@ca2ddb owned by: Blueprint Extender: 1 Total blocked: 9 Total waited: 12 Stack trace: org.apache.aries.blueprint.container.BlueprintContainerImpl.destroy(BlueprintContainerImpl.java:842) org.apache.aries.blueprint.container.BlueprintExtender$3.run(BlueprintExtender.java:284) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) org.apache.aries.blueprint.container.BlueprintExtender.destroyContainer(BlueprintExtender.java:305) org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:206) org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500) org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433) org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725) org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463) org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422) org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103) org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695) org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483) org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4244) org.apache.felix.framework.Felix.stopBundle(Felix.java:2351) org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4629) org.apache.felix.framework.Felix.refreshPackages(Felix.java:3951) org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:172) java.lang.Thread.run(Thread.java:722)
          Hide
          wverreydt Wim Verreydt added a comment -

          I also have this issue. When I put a dev:watch on a bundle (even tested with a empty blueprint file), karaf will try to stop the bundle but never succeed. I'll attach my jstack output. I would be very happy if this could be solved, this issue really slows down development.

          Show
          wverreydt Wim Verreydt added a comment - I also have this issue. When I put a dev:watch on a bundle (even tested with a empty blueprint file), karaf will try to stop the bundle but never succeed. I'll attach my jstack output. I would be very happy if this could be solved, this issue really slows down development.
          Hide
          dnk Dmitry Konstantinov added a comment - - edited

          I have the simular case during Karaf startup, it's looks like a deadlock in Blueprint container. I have created the issue: https://issues.apache.org/jira/browse/ARIES-1051

          Show
          dnk Dmitry Konstantinov added a comment - - edited I have the simular case during Karaf startup, it's looks like a deadlock in Blueprint container. I have created the issue: https://issues.apache.org/jira/browse/ARIES-1051

            People

            • Assignee:
              Unassigned
              Reporter:
              ceefour Hendy Irawan
            • Votes:
              4 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development