ActiveMQ
  1. ActiveMQ
  2. AMQ-3681

DatabaseLocker should first cancel locking SQL statement before closing the SQL connection

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.4.2
    • Fix Version/s: 5.6.0
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      ServiceMix 4.3

    • Patch Info:
      Patch Available

      Description

      ActiveMQ is configured in a Master/Slave configuration with an Oracle database :
      http://activemq.apache.org/jdbc-master-slave.html
      http://servicemix.apache.org/clustering.html
      When the slave node is stopping, "activemq-broker" stays forever in the "Stopping" state.
      This is because the locking SQL statement cannot be interrupted by just closing the JDBC connection. It is also needed to "cancel" the SQL statement.
      Here is a patch to DefaultDatabaseLocker which makes it compatible with Oracle.
      Thanks.

      
      "Thread-92" prio=10 tid=0x08c4d800 nid=0x1036 waiting for monitor entry [0x8ab3a000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at oracle.jdbc.driver.PhysicalConnection.isClosed(PhysicalConnection.java:1223)
      	- waiting to lock <0xad4367e0> (a oracle.jdbc.driver.T4CConnection)
      	at org.apache.commons.dbcp.DelegatingConnection.isClosed(DelegatingConnection.java:386)
      	at org.apache.commons.dbcp.DelegatingConnection.isClosed(DelegatingConnection.java:386)
      	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.isClosed(PoolingDataSource.java:201)
      	at org.apache.activemq.store.jdbc.DefaultDatabaseLocker.stop(DefaultDatabaseLocker.java:137)
      	at com.mycompany.PoolCloser.close(PoolCloser.java:77)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:221)
      	at org.apache.aries.blueprint.container.ServiceListener.invokeMethod(ServiceListener.java:98)
      	at org.apache.aries.blueprint.container.ServiceListener.unregister(ServiceListener.java:65)
      
      1. amq_stopping_slave.patch
        2 kB
        metatech
      2. amq_stopping_slave_2.patch
        4 kB
        metatech

        Issue Links

          Activity

          Hide
          metatech added a comment -

          Another problem that can occur :
          when the network cable is unplugged, the ActiveMQ shutdown hook freezes because it cannot close the connection, which is locked by the "DefaultDatabaseLocker.keepAlive" method. The "kill" command does not work, "kill -9" is needed.
          Here is a second patch that also fixes this problem.

          "ActiveMQ ShutdownHook" daemon prio=5 Thread id=73 BLOCKED
          	oracle.jdbc.driver.PhysicalConnection.isClosed(PhysicalConnection.java:1223)
          	org.apache.commons.dbcp.DelegatingConnection.isClosed(DelegatingConnection.java:386)
          	org.apache.commons.dbcp.DelegatingConnection.isClosed(DelegatingConnection.java:386)
          	org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.isClosed(PoolingDataSource.java:201)
          	org.apache.activemq.store.jdbc.DefaultDatabaseLocker.stop(DefaultDatabaseLocker.java:137)
          	org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.stop(JDBCPersistenceAdapter.java:328)
          	org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
          	org.apache.activemq.broker.BrokerService.stop(BrokerService.java:583)
          	org.apache.activemq.broker.BrokerService.containerShutdown(BrokerService.java:1971)
          	org.apache.activemq.broker.BrokerService$4.run(BrokerService.java:1938)
          
          Show
          metatech added a comment - Another problem that can occur : when the network cable is unplugged, the ActiveMQ shutdown hook freezes because it cannot close the connection, which is locked by the "DefaultDatabaseLocker.keepAlive" method. The "kill" command does not work, "kill -9" is needed. Here is a second patch that also fixes this problem. "ActiveMQ ShutdownHook" daemon prio=5 Thread id=73 BLOCKED oracle.jdbc.driver.PhysicalConnection.isClosed(PhysicalConnection.java:1223) org.apache.commons.dbcp.DelegatingConnection.isClosed(DelegatingConnection.java:386) org.apache.commons.dbcp.DelegatingConnection.isClosed(DelegatingConnection.java:386) org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.isClosed(PoolingDataSource.java:201) org.apache.activemq.store.jdbc.DefaultDatabaseLocker.stop(DefaultDatabaseLocker.java:137) org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.stop(JDBCPersistenceAdapter.java:328) org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41) org.apache.activemq.broker.BrokerService.stop(BrokerService.java:583) org.apache.activemq.broker.BrokerService.containerShutdown(BrokerService.java:1971) org.apache.activemq.broker.BrokerService$4.run(BrokerService.java:1938)
          Hide
          Rob Davies added a comment -

          patch applied in SVN revision 1325621

          Show
          Rob Davies added a comment - patch applied in SVN revision 1325621
          Hide
          Gary Tully added a comment - - edited

          Unfortunately the embedded derby jdbc store used in the unit tests does not support java.sql.Statement#setQueryTimeout

          We need to use it in an oracle specific lock implementation of make its use configurable

          org.apache.activemq.usecases.JdbcDurableSubDupTest demonstrates

          Show
          Gary Tully added a comment - - edited Unfortunately the embedded derby jdbc store used in the unit tests does not support java.sql.Statement#setQueryTimeout We need to use it in an oracle specific lock implementation of make its use configurable org.apache.activemq.usecases.JdbcDurableSubDupTest demonstrates
          Hide
          Gary Tully added a comment -

          hmm. well derby should have it once we upgrade[1], can any one easily validate mysql or mssqlserver?

          [1] https://issues.apache.org/jira/browse/DERBY-31

          Show
          Gary Tully added a comment - hmm. well derby should have it once we upgrade [1] , can any one easily validate mysql or mssqlserver? [1] https://issues.apache.org/jira/browse/DERBY-31
          Hide
          Gary Tully added a comment -

          junit tests are ok with the latest derby and looks like current versions of mysql and mssqlserver have support but not postgresql.
          Making this conditional on queryTimeout attribute > 0 to that it can be disabled via configuration on the default database locker. Default value 10seconds.

          http://svn.apache.org/viewvc?rev=1326610&view=rev

          Show
          Gary Tully added a comment - junit tests are ok with the latest derby and looks like current versions of mysql and mssqlserver have support but not postgresql. Making this conditional on queryTimeout attribute > 0 to that it can be disabled via configuration on the default database locker. Default value 10seconds. http://svn.apache.org/viewvc?rev=1326610&view=rev

            People

            • Assignee:
              Rob Davies
              Reporter:
              metatech
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development