Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-8140

[Broker-J][BDB HA] Removal of non existing group member can end up in broker crash due to uncaught MemberNotFoundException

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: qpid-java-broker-7.0.2, qpid-java-broker-7.0.0, qpid-java-broker-7.0.1
    • Fix Version/s: qpid-java-broker-7.1.0
    • Component/s: Broker-J
    • Labels:
      None

      Description

      When two concurrent requests are made to delete a group member using REST API for remote replication node, one can successfully remove the node, whilst other can end-up in un-handled  MemberNotFoundException which crashes the broker.

      An example of remote node delete request

       curl -v -k -u admin -X DELETE  http://localhost:8080/api/latest/remotereplicationnode/node2/node1
      

      For failed requests an exception like the one below is reported

       ERROR [VirtualHostNode-node2-Config] o.a.q.s.u.ServerScopedRuntimeException Exception on node removal from group
      com.sleepycat.je.rep.MemberNotFoundException: (JE 7.4.5) Node: node1 is not currently a member of the group: BDBHAVirtualHostNodeRestTest.testDeleteMasterNode, it has been removed.
      	at com.sleepycat.je.rep.util.ReplicationGroupAdmin.checkMember(ReplicationGroupAdmin.java:576)
      	at com.sleepycat.je.rep.util.ReplicationGroupAdmin.removeMember(ReplicationGroupAdmin.java:314)
      	at org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.removeNodeFromGroup(ReplicatedEnvironmentFacade.java:1210)
      	at org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHARemoteReplicationNodeImpl.onDelete(BDBHARemoteReplicationNodeImpl.java:139)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2759)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:360)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
      	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
      	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
      	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
      	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
      	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:822)
      	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:664)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$ChainedSettableFuture.set(AbstractConfiguredObject.java:2710)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$21$1.onSuccess(AbstractConfiguredObject.java:2765)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:360)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
      	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
      	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
      	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
      	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
      	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
      	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
      	at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
      	at org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2884)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2760)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:360)
      	at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
      	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
      	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:415)
      	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:411)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:360)
      	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1.run(TaskExecutorImpl.java:410)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
      	at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Activity

            People

            • Assignee:
              orudyy Alex Rudyy
              Reporter:
              orudyy Alex Rudyy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: