Uploaded image for project: 'Karaf'
  1. Karaf
  2. KARAF-1925

Blueprint timeout exceptions after karaf startup

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.2
    • Component/s: karaf-core
    • Labels:
      None

      Description

      the karaf management bundle (or the karaf ssh bundle) sometimes will be on the GracePeriod after start karaf container.

      [ 17] [Active ] [GracePeriod ] [ ] [ 30] Apache Karaf :: Shell :: SSH (2.3.0)
      or
      [ 18] [Active ] [GracePeriod ] [ ] [ 30] Apache Karaf :: Management (2.3.0)

      and got the blueprint timeout exceptions in the karaf.log (attached)

      if then refresh the bundles manually:

      karaf@root>refresh 18
      or
      karaf@root>refresh 17

      then, everything will be ok.

      1. AfterFailThreadDump.txt
        39 kB
        Pavel Vasilchenko
      2. FailLog.txt
        48 kB
        Pavel Vasilchenko
      3. GracePeriodBeforeFailThreadDump.txt
        40 kB
        Pavel Vasilchenko
      4. karaf.log
        64 kB
        Xilai Dai
      5. NormalStartupThreadDump.txt
        48 kB
        Pavel Vasilchenko

        Issue Links

          Activity

          Hide
          xldai Xilai Dai added a comment -

          the only thing I did change is set value of respectStartLvlDuringFeatureStartup property from false to true in the etc/org.apache.karaf.features.cfg, but not sure if this issue exists when respectStartLvlDuringFeatureStartup=false (default value)

          Show
          xldai Xilai Dai added a comment - the only thing I did change is set value of respectStartLvlDuringFeatureStartup property from false to true in the etc/org.apache.karaf.features.cfg, but not sure if this issue exists when respectStartLvlDuringFeatureStartup=false (default value)
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          Thanks for the update, the respectStartLvlDuringFeatureStartup property is interesting. Let me try to reproduce like this.

          Show
          jbonofre Jean-Baptiste Onofré added a comment - Thanks for the update, the respectStartLvlDuringFeatureStartup property is interesting. Let me try to reproduce like this.
          Hide
          pieber Andreas Pieber added a comment -

          OK, I know that I've tested this before pushing At least I can confirm that it works perfectly on ...

          ➜  ~  uname -a
          Linux prime 3.6.2-1-ARCH #1 SMP PREEMPT Fri Oct 12 23:58:58 CEST 2012 x86_64 GNU/Linux
          

          ...and...

          ➜  ~  java -version
          java version "1.7.0_07"
          OpenJDK Runtime Environment (IcedTea7 2.3.2) (ArchLinux build 7.u7_2.3.2-2-x86_64)
          OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
          

          ...or...

          ➜  ~  /opt/java6/bin/java -version
          java version "1.6.0_35"
          Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
          Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)
          

          I've no windows 7 at hand right now (limited space on ssd ) but I'll try on my XP VM with Java 7. Let's hope I can reproduce your problem there...

          Show
          pieber Andreas Pieber added a comment - OK, I know that I've tested this before pushing At least I can confirm that it works perfectly on ... ➜ ~ uname -a Linux prime 3.6.2-1-ARCH #1 SMP PREEMPT Fri Oct 12 23:58:58 CEST 2012 x86_64 GNU/Linux ...and... ➜ ~ java -version java version "1.7.0_07" OpenJDK Runtime Environment (IcedTea7 2.3.2) (ArchLinux build 7.u7_2.3.2-2-x86_64) OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode) ...or... ➜ ~ /opt/java6/bin/java -version java version "1.6.0_35" Java(TM) SE Runtime Environment (build 1.6.0_35-b10) Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode) I've no windows 7 at hand right now (limited space on ssd ) but I'll try on my XP VM with Java 7. Let's hope I can reproduce your problem there...
          Hide
          pieber Andreas Pieber added a comment -

          Windows XP, SP2, all patches installed with...

          C:\> java -version
          java version "1.7.0_07"
          Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
          Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
          

          and it works like a charm too...

          I give up. Any more information you can provide us with? Maybe it's a timing issue? Can you try if you get the same result if you set respectStartLvlDuringFeatureStartup=false (the default value)?

          Thanks

          Show
          pieber Andreas Pieber added a comment - Windows XP, SP2, all patches installed with... C:\> java -version java version "1.7.0_07" Java(TM) SE Runtime Environment (build 1.7.0_07-b11) Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode) and it works like a charm too... I give up. Any more information you can provide us with? Maybe it's a timing issue? Can you try if you get the same result if you set respectStartLvlDuringFeatureStartup=false (the default value)? Thanks
          Hide
          xldai Xilai Dai added a comment -

          Hi, Andreas, I can't reproduce this issue with respectStartLvlDuringFeatureStartup=false after about 30 times restart container. but with respectStartLvlDuringFeatureStartup=true, the exceptions will be shown after about 10-20 times restart container.

          Show
          xldai Xilai Dai added a comment - Hi, Andreas, I can't reproduce this issue with respectStartLvlDuringFeatureStartup=false after about 30 times restart container. but with respectStartLvlDuringFeatureStartup=true, the exceptions will be shown after about 10-20 times restart container.
          Hide
          pieber Andreas Pieber added a comment -

          maybe my computer is simply too fast... are you using 2.3.0 (FINAL) or the 2.3.1-SNAPSHOTs? If this is the case, can you try to increase the start-lvl of those two bundles a bit (or decrease it) and check if the error still occurs. I would do it myself if I could reproduce it, but so I woult really appreciate your help.

          Thank you very much!

          Show
          pieber Andreas Pieber added a comment - maybe my computer is simply too fast... are you using 2.3.0 (FINAL) or the 2.3.1-SNAPSHOTs? If this is the case, can you try to increase the start-lvl of those two bundles a bit (or decrease it) and check if the error still occurs. I would do it myself if I could reproduce it, but so I woult really appreciate your help. Thank you very much!
          Hide
          xldai Xilai Dai added a comment -

          Hi, I'm using 2.3.0 release.

          I've tried to increase the start-lv of ssh and management bundle from 30 to 40 in the standard-2.3.0-features.xml, like this:
          <feature name="ssh" description="Provide a SSHd server on Karaf" version="2.3.0">
          ...
          <bundle start-level="40">mvn:org.apache.karaf.shell/org.apache.karaf.shell.ssh/2.3.0</bundle>
          </feature>
          <feature name="management" description="Provide a JMX MBeanServer and a set of MBeans in Karaf" version="2.3.0">
          <bundle start-level="40">mvn:org.apache.karaf.management/org.apache.karaf.management.server/2.3.0</bundle>
          ...
          </feature>

          with these changes, it's really easy to reproduce this problem after those bundles failed , you will got the blueprint timeout exceptions.
          [ 47] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: System (2.3.0)
          [ 48] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: Bundles (2.3.0)
          [ 49] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: Services (2.3.0)
          [ 50] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: Config (2.3.0)
          [ 51] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: Log (2.3.0)

          Show
          xldai Xilai Dai added a comment - Hi, I'm using 2.3.0 release. I've tried to increase the start-lv of ssh and management bundle from 30 to 40 in the standard-2.3.0-features.xml, like this: <feature name="ssh" description="Provide a SSHd server on Karaf" version="2.3.0"> ... <bundle start-level="40">mvn:org.apache.karaf.shell/org.apache.karaf.shell.ssh/2.3.0</bundle> </feature> <feature name="management" description="Provide a JMX MBeanServer and a set of MBeans in Karaf" version="2.3.0"> <bundle start-level="40">mvn:org.apache.karaf.management/org.apache.karaf.management.server/2.3.0</bundle> ... </feature> with these changes, it's really easy to reproduce this problem after those bundles failed , you will got the blueprint timeout exceptions. [ 47] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: System (2.3.0) [ 48] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: Bundles (2.3.0) [ 49] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: Services (2.3.0) [ 50] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: Config (2.3.0) [ 51] [Active ] [Failure ] [ 30] Apache Karaf :: Management :: MBeans :: Log (2.3.0)
          Hide
          xldai Xilai Dai added a comment -

          so, maybe decrease start-lv of ssh/management from 30 to 25 will be a workaround with respectStartLvlDuringFeatureStartup=true case, need more test...

          Show
          xldai Xilai Dai added a comment - so, maybe decrease start-lv of ssh/management from 30 to 25 will be a workaround with respectStartLvlDuringFeatureStartup=true case, need more test...
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          It depends the "dependencies". So basically, it means that the issue could be in the feature descriptor itself (start level) and not in the FeaturesService directly.

          Show
          jbonofre Jean-Baptiste Onofré added a comment - It depends the "dependencies". So basically, it means that the issue could be in the feature descriptor itself (start level) and not in the FeaturesService directly.
          Hide
          pavel.vasilchenko Pavel Vasilchenko added a comment -

          Hi, I also have reproduced this issue even with respectStartLvlDuringFeatureStartup=false.
          Also I have created thread dumps with jvisualvm and attached file to this issue.
          Maybe it will help you.

          Show
          pavel.vasilchenko Pavel Vasilchenko added a comment - Hi, I also have reproduced this issue even with respectStartLvlDuringFeatureStartup=false. Also I have created thread dumps with jvisualvm and attached file to this issue. Maybe it will help you.
          Hide
          xldai Xilai Dai added a comment -

          FYI, can't reproduce this issue in case respectStartLvlDuringFeatureStartup=true, but add dependency="true" property to:
          <bundle start-level="30" dependency="true">mvn:org.apache.karaf.management/org.apache.karaf.management.server/2.3.0</bundle>
          in the management feature of feature descriptor.

          Show
          xldai Xilai Dai added a comment - FYI, can't reproduce this issue in case respectStartLvlDuringFeatureStartup=true, but add dependency="true" property to: <bundle start-level="30" dependency="true">mvn:org.apache.karaf.management/org.apache.karaf.management.server/2.3.0</bundle> in the management feature of feature descriptor.
          Hide
          xldai Xilai Dai added a comment -

          Also can't reproduce this issue in case you add :

          org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.system/2.3.0/org.apache.karaf.management.mbeans.system-2.3.0.jar=30
          org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.bundles/2.3.0/org.apache.karaf.management.mbeans.bundles-2.3.0.jar=30
          org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.services/2.3.0/org.apache.karaf.management.mbeans.services-2.3.0.jar=30
          org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.config/2.3.0/org.apache.karaf.management.mbeans.config-2.3.0.jar=30
          org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.log/2.3.0/org.apache.karaf.management.mbeans.log-2.3.0.jar=30
          org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.packages/2.3.0/org.apache.karaf.management.mbeans.packages-2.3.0.jar=30
          org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.dev/2.3.0/org.apache.karaf.management.mbeans.dev-2.3.0.jar=30

          to the startup.properties (under the line of org/apache/aries/jmx/org.apache.aries.jmx.blueprint.core/1.0.1/org.apache.aries.jmx.blueprint.core-1.0.1.jar=30 )

          Show
          xldai Xilai Dai added a comment - Also can't reproduce this issue in case you add : org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.system/2.3.0/org.apache.karaf.management.mbeans.system-2.3.0.jar=30 org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.bundles/2.3.0/org.apache.karaf.management.mbeans.bundles-2.3.0.jar=30 org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.services/2.3.0/org.apache.karaf.management.mbeans.services-2.3.0.jar=30 org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.config/2.3.0/org.apache.karaf.management.mbeans.config-2.3.0.jar=30 org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.log/2.3.0/org.apache.karaf.management.mbeans.log-2.3.0.jar=30 org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.packages/2.3.0/org.apache.karaf.management.mbeans.packages-2.3.0.jar=30 org/apache/karaf/management/mbeans/org.apache.karaf.management.mbeans.dev/2.3.0/org.apache.karaf.management.mbeans.dev-2.3.0.jar=30 to the startup.properties (under the line of org/apache/aries/jmx/org.apache.aries.jmx.blueprint.core/1.0.1/org.apache.aries.jmx.blueprint.core-1.0.1.jar=30 )
          Hide
          pavel.vasilchenko Pavel Vasilchenko added a comment -

          I reproduced bug with this configuration

          Show
          pavel.vasilchenko Pavel Vasilchenko added a comment - I reproduced bug with this configuration
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          The following changes in etc/startup.properties look to provide good result:

          org/apache/karaf/features/org.apache.karaf.features.core/2.3.0/org.apache.karaf.features.core-2.3.0.jar=25

          org/apache/karaf/jaas/org.apache.karaf.jaas.config/2.3.0/org.apache.karaf.jaas.config-2.3.0.jar=25
          org/apache/karaf/jaas/org.apache.karaf.jaas.modules/2.3.0/org.apache.karaf.jaas.modules-2.3.0.jar=25
          org/apache/karaf/jaas/org.apache.karaf.jaas.command/2.3.0/org.apache.karaf.jaas.command-2.3.0.jar=25

          In the mean time, as I suspect a "low" timeout in the Blueprint service reference resolution, I try to increase this timeout.

          Show
          jbonofre Jean-Baptiste Onofré added a comment - The following changes in etc/startup.properties look to provide good result: org/apache/karaf/features/org.apache.karaf.features.core/2.3.0/org.apache.karaf.features.core-2.3.0.jar=25 org/apache/karaf/jaas/org.apache.karaf.jaas.config/2.3.0/org.apache.karaf.jaas.config-2.3.0.jar=25 org/apache/karaf/jaas/org.apache.karaf.jaas.modules/2.3.0/org.apache.karaf.jaas.modules-2.3.0.jar=25 org/apache/karaf/jaas/org.apache.karaf.jaas.command/2.3.0/org.apache.karaf.jaas.command-2.3.0.jar=25 In the mean time, as I suspect a "low" timeout in the Blueprint service reference resolution, I try to increase this timeout.
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          I try with:

          Bundle-SymbolicName: foobar;
          blueprint.graceperiod:=false

          Show
          jbonofre Jean-Baptiste Onofré added a comment - I try with: Bundle-SymbolicName: foobar; blueprint.graceperiod:=false
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          I also add timeout=0.

          Show
          jbonofre Jean-Baptiste Onofré added a comment - I also add timeout=0.
          Hide
          chris@die-schneider.net Christian Schneider added a comment -

          I can reproduce the issue on karaf 2.3.0 by doing the following:
          download and extract fresh karaf 2.3.0
          bin/karaf
          ^D
          bin/karaf
          la -s | grep ssh
          [ 17] [Active ] [GracePeriod ] [ 30] org.apache.karaf.shell.ssh (2.3.0)
          [ 24] [Active ] [ ] [ 30] org.apache.sshd.core (0.8.0)

          I can not reproduce with current 2.3.x snapshot

          Show
          chris@die-schneider.net Christian Schneider added a comment - I can reproduce the issue on karaf 2.3.0 by doing the following: download and extract fresh karaf 2.3.0 bin/karaf ^D bin/karaf la -s | grep ssh [ 17] [Active ] [GracePeriod ] [ 30] org.apache.karaf.shell.ssh (2.3.0) [ 24] [Active ] [ ] [ 30] org.apache.sshd.core (0.8.0) I can not reproduce with current 2.3.x snapshot
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          blueprint.graceperiod:=false works.

          I think it's an issue in Aries Blueprint 1.0.1. Let me create a Jira there.

          Show
          jbonofre Jean-Baptiste Onofré added a comment - blueprint.graceperiod:=false works. I think it's an issue in Aries Blueprint 1.0.1. Let me create a Jira there.
          Hide
          ceefour Hendy Irawan added a comment - - edited

          Please change the environment to All, as this is not specific to Windows 7 (see #KARAF-1931).

          Our environment is:

          Karaf
          Karaf version 2.3.0
          Karaf home /home/rudi/git/bippo-commerce5/karaf
          Karaf base /home/rudi/git/bippo-commerce5/karaf
          OSGi Framework org.apache.felix.framework - 4.0.3
          JVM
          Java Virtual Machine OpenJDK 64-Bit Server VM version 23.2-b09
          Version 1.7.0_07
          Vendor Oracle Corporation
          Uptime 15.401 seconds
          Total compile time 27.499 seconds
          Threads
          Live threads 113
          Daemon threads 98
          Peak 115
          Total started 136
          Memory
          Current heap size 75,670 kbytes
          Maximum heap size 466,048 kbytes
          Committed heap size 245,888 kbytes
          Pending objects 0
          Garbage collector Name = 'PS Scavenge', Collections = 22, Time = 0.221 seconds
          Garbage collector Name = 'PS MarkSweep', Collections = 1, Time = 0.130 seconds
          Classes
          Current classes loaded 9,043
          Total classes loaded 9,044
          Total classes unloaded 1
          Operating system
          Name Linux version 3.2.0-32-generic
          Architecture amd64
          Processors 4
          
          Show
          ceefour Hendy Irawan added a comment - - edited Please change the environment to All, as this is not specific to Windows 7 (see # KARAF-1931 ). Our environment is: Karaf Karaf version 2.3.0 Karaf home /home/rudi/git/bippo-commerce5/karaf Karaf base /home/rudi/git/bippo-commerce5/karaf OSGi Framework org.apache.felix.framework - 4.0.3 JVM Java Virtual Machine OpenJDK 64-Bit Server VM version 23.2-b09 Version 1.7.0_07 Vendor Oracle Corporation Uptime 15.401 seconds Total compile time 27.499 seconds Threads Live threads 113 Daemon threads 98 Peak 115 Total started 136 Memory Current heap size 75,670 kbytes Maximum heap size 466,048 kbytes Committed heap size 245,888 kbytes Pending objects 0 Garbage collector Name = 'PS Scavenge', Collections = 22, Time = 0.221 seconds Garbage collector Name = 'PS MarkSweep', Collections = 1, Time = 0.130 seconds Classes Current classes loaded 9,043 Total classes loaded 9,044 Total classes unloaded 1 Operating system Name Linux version 3.2.0-32- generic Architecture amd64 Processors 4
          Hide
          gcollins Gareth Collins added a comment -

          I see this problem running ActiveMQ 5.7 in karaf. Sometimes broker blueprint files are getting stuck in "Grace Period" state.

          Show
          gcollins Gareth Collins added a comment - I see this problem running ActiveMQ 5.7 in karaf. Sometimes broker blueprint files are getting stuck in "Grace Period" state.
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          I think that I addressed this issue for Karaf 2.3.2 by adding org.apache.aries.blueprint.synchronous=true in etc/config.properties.

          Show
          jbonofre Jean-Baptiste Onofré added a comment - I think that I addressed this issue for Karaf 2.3.2 by adding org.apache.aries.blueprint.synchronous=true in etc/config.properties.
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          I close this issue as it should be fixed with blueprint.synchronous=true. Please, reopen the Jira if you see this issue again.

          Show
          jbonofre Jean-Baptiste Onofré added a comment - I close this issue as it should be fixed with blueprint.synchronous=true. Please, reopen the Jira if you see this issue again.
          Hide
          ceefour666@gmail.com Hendy Irawan added a comment -

          Thank you Jean-Baptiste!
          On Jul 2, 2013 11:51 PM, "Jean-Baptiste Onofré (JIRA)" <jira@apache.org>

          Show
          ceefour666@gmail.com Hendy Irawan added a comment - Thank you Jean-Baptiste! On Jul 2, 2013 11:51 PM, "Jean-Baptiste Onofré (JIRA)" <jira@apache.org>
          Hide
          skyblender Marcos added a comment -

          Does this setting work on Karaf 2.3.1?

          Show
          skyblender Marcos added a comment - Does this setting work on Karaf 2.3.1?

            People

            • Assignee:
              jbonofre Jean-Baptiste Onofré
              Reporter:
              xldai Xilai Dai
            • Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development