Major Description
I have three karaf nodes running activemq and camel. Activemq is configured for JDBC Master/Slave on all three nodes.
scenario:
node1a is the AMQ Master and node1b and node2a are slaves. Everything working great until you try and shutdown node1b or node2a.
When the shutdown is triggered, the activemq-osgi bundle stays in a "stopping" state until it can obtain the master, then continues to shutdown - ex: forever.
Attached are the activemq.xml configuration files for the three nodes.
Note also that this same behavior happens on NON-Camel nodes (node2a is only karaf and amq).
I triggered a shutdown on node1b and here is the status of activemq:
113 | Active | 50 | 5.13.3 | activemq-karaf 152 | Stopping | 50 | 5.13.3 | activemq-osgi 368 | Resolved | 50 | 5.13.3 | activemq-camel
Turned up logging on AMQ, here is the log entries that keep repeating after "shutdown -f" on the slave node:
20171024 12:37:08.922 [DEBUG] FelixStartLevel | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.activemq-osgi | ServiceEvent UNREGISTERING - [org.apache.camel.spi.ComponentResolver] - org.apache.activemq.activemq-osgi 20171024 12:37:08.923 [DEBUG] FelixStartLevel | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.activemq-osgi | ServiceEvent UNREGISTERING - [org.apache.camel.spi.TypeConverterLoader] - org.apache.activemq.activemq-osgi 20171024 12:37:08.924 [DEBUG] FelixStartLevel | 368:org.apache.activemq.activemq-camel | org.apache.activemq.activemq-camel | ServiceEvent UNREGISTERING - [org.apache.camel.spi.ComponentResolver] - org.apache.activemq.activemq-camel 20171024 12:37:08.925 [DEBUG] FelixStartLevel | 368:org.apache.activemq.activemq-camel | org.apache.activemq.activemq-camel | ServiceEvent UNREGISTERING - [org.apache.camel.spi.TypeConverterLoader] - org.apache.activemq.activemq-camel 20171024 12:37:08.925 [INFO ] FelixStartLevel | 372:org.apache.camel.camel-core | org.apache.camel.impl.osgi.Activator | Camel activator stopped 20171024 12:37:08.926 [INFO ] FelixStartLevel | 23:org.apache.aries.blueprint.core | org.apache.aries.blueprint.container.BlueprintExtender | Destroying BlueprintContainer for bundle org.apache.camel.camel-blueprint/2.17.4 20171024 12:37:08.930 [DEBUG] FelixStartLevel | 368:org.apache.activemq.activemq-camel | org.apache.activemq.activemq-camel | BundleEvent STOPPING - org.apache.activemq.activemq-camel 20171024 12:37:08.931 [DEBUG] FelixStartLevel | 368:org.apache.activemq.activemq-camel | org.apache.activemq.activemq-camel | BundleEvent STOPPED - org.apache.activemq.activemq-camel 20171024 12:37:08.935 [INFO ] FelixStartLevel | 23:org.apache.aries.blueprint.core | org.apache.aries.blueprint.container.BlueprintExtender | Destroying BlueprintContainer for bundle org.apache.activemq.activemq-osgi/5.13.3 20171024 12:37:08.937 [DEBUG] FelixStartLevel | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.activemq-osgi | ServiceEvent UNREGISTERING - [org.osgi.service.blueprint.container.BlueprintContainer] - org.apache.activemq.activemq-osgi 20171024 12:37:08.938 [DEBUG] FelixStartLevel | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.activemq-osgi | ServiceEvent UNREGISTERING - [org.osgi.service.cm.ManagedServiceFactory] - org.apache.activemq.activemq-osgi 20171024 12:37:10.920 [DEBUG] ActiveMQ InactivityMonitor ReadCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | 30000ms elapsed since last read check. 20171024 12:37:10.920 [TRACE] ActiveMQ InactivityMonitor ReadCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | A receive is in progress, skipping read check. 20171024 12:37:11.033 [DEBUG] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | WriteChecker: 10001ms elapsed since last write check. 20171024 12:37:11.033 [TRACE] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | tcp://localhost/127.0.0.1:8206@52881 no message sent since last write check, sending a KeepAliveInfo 20171024 12:37:11.033 [DEBUG] ActiveMQ InactivityMonitor Worker | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | Running WriteCheck[tcp://127.0.0.1:8206] 20171024 12:37:21.033 [DEBUG] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | WriteChecker: 10000ms elapsed since last write check. 20171024 12:37:21.033 [TRACE] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | tcp://localhost/127.0.0.1:8206@52881 no message sent since last write check, sending a KeepAliveInfo 20171024 12:37:21.034 [DEBUG] ActiveMQ InactivityMonitor Worker | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | Running WriteCheck[tcp://127.0.0.1:8206] 20171024 12:37:31.033 [DEBUG] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | WriteChecker: 10000ms elapsed since last write check. 20171024 12:37:31.033 [TRACE] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | tcp://localhost/127.0.0.1:8206@52881 no message sent since last write check, sending a KeepAliveInfo 20171024 12:37:31.033 [DEBUG] ActiveMQ InactivityMonitor Worker | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | Running WriteCheck[tcp://127.0.0.1:8206] 20171024 12:37:40.921 [DEBUG] ActiveMQ InactivityMonitor ReadCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | 30001ms elapsed since last read check. 20171024 12:37:40.921 [TRACE] ActiveMQ InactivityMonitor ReadCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | A receive is in progress, skipping read check. 20171024 12:37:41.033 [DEBUG] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | WriteChecker: 10000ms elapsed since last write check. 20171024 12:37:41.033 [TRACE] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | tcp://localhost/127.0.0.1:8206@52881 no message sent since last write check, sending a KeepAliveInfo 20171024 12:37:41.034 [DEBUG] ActiveMQ InactivityMonitor Worker | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | Running WriteCheck[tcp://127.0.0.1:8206] 20171024 12:37:51.033 [DEBUG] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | WriteChecker: 10000ms elapsed since last write check. 20171024 12:37:51.033 [TRACE] ActiveMQ InactivityMonitor WriteCheckTimer | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | tcp://localhost/127.0.0.1:8206@52881 no message sent since last write check, sending a KeepAliveInfo 20171024 12:37:51.033 [DEBUG] ActiveMQ InactivityMonitor Worker | 152:org.apache.activemq.activemq-osgi | org.apache.activemq.transport.AbstractInactivityMonitor | Running WriteCheck[tcp://127.0.0.1:8206]