[QPID-8140] [Broker-J][BDB HA] Removal of non existing group member can end up in broker crash due to uncaught MemberNotFoundException

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: qpid-java-broker-7.0.2, qpid-java-broker-7.0.0, qpid-java-broker-7.0.1
• Fix Version/s: qpid-java-broker-7.1.0
• Component/s: Broker-J
• Assignee: Alex Rudyy
• Reporter: Alex Rudyy
• Created: 27/Mar/18 09:39
• Updated: 05/Jan/19 20:01
• Resolved: 29/Mar/18 22:02

Description

When two concurrent requests are made to delete a group member using REST API for remote replication node, one can successfully remove the node, whilst other can end-up in un-handled  MemberNotFoundException which crashes the broker.

An example of remote node delete request

 curl -v -k -u admin -X DELETE  http://localhost:8080/api/latest/remotereplicationnode/node2/node1

For failed requests an exception like the one below is reported

 ERROR [VirtualHostNode-node2-Config] o.a.q.s.u.ServerScopedRuntimeException Exception on node removal from group
com.sleepycat.je.rep.MemberNotFoundException: (JE 7.4.5) Node: node1 is not currently a member of the group: BDBHAVirtualHostNodeRestTest.testDeleteMasterNode, it has been removed.
	at com.sleepycat.je.rep.util.ReplicationGroupAdmin.checkMember(ReplicationGroupAdmin.java:576)
	at com.sleepycat.je.rep.util.ReplicationGroupAdmin.removeMember(ReplicationGroupAdmin.java:314)
	at org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.removeNodeFromGroup(ReplicatedEnvironmentFacade.java:1210)
	at org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHARemoteReplicationNodeImpl.onDelete(BDBHARemoteReplicationNodeImpl.java:139)
	at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2759)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:822)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:664)
	at org.apache.qpid.server.model.AbstractConfiguredObject$ChainedSettableFuture.set(AbstractConfiguredObject.java:2710)
	at org.apache.qpid.server.model.AbstractConfiguredObject$21$1.onSuccess(AbstractConfiguredObject.java:2765)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
	at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
	at org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2884)
	at org.apache.qpid.server.model.AbstractConfiguredObject$21.onSuccess(AbstractConfiguredObject.java:2760)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2894)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24$1.run(AbstractConfiguredObject.java:2890)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.qpid.server.model.AbstractConfiguredObject$24.onSuccess(AbstractConfiguredObject.java:2889)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:415)
	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1$1.run(TaskExecutorImpl.java:411)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor$1.run(TaskExecutorImpl.java:410)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
	at java.lang.Thread.run(Thread.java:748)