[QPID-6560] [Java Broker] BDB HA JE environment close on intruder detection might block the execution of VHN children tasks thus causing unnecessary delays in shutdown of ReplicatedEnvironmentFacade executors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.32
• Fix Version/s: qpid-java-6.0
• Component/s: Broker-J
• Assignee: Alex Rudyy
• Reporter: Alex Rudyy
• Created: 29/May/15 13:39
• Updated: 14/Mar/17 21:29
• Resolved: 28/Sep/15 22:17

Description

On intruder detection a task to close VHN children and set VHN state to ERRORED is scheduled in Broker configuration thread. Immediately after scheduling the task, ReplicatedEnvironmentFacade.close() is invoked.
ReplicatedEnvironmentFacade executors are shutdown in close method.
If any of ReplicatedEnvironmentFacade executors has a pending work (tasks to run) and that work needs to be performed in VHN configuration thread or Broker configuration thread in synchronous manner (blocking ReplicatedEnvironmentFacade executors threads), the executors shutdown would be blocked and eventually times out.

Test BDBHAVirtualHostNodeRestTest.testIntruderProtection fails sporadically as indicated by stack trace below:

junit.framework.AssertionFailedError: Attribute state did not reach expected value within permitted timeout 5000ms. expected:<ERRORED> but was:<ACTIVE>
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.TestCase.assertEquals(TestCase.java:244)
	at org.apache.qpid.systest.rest.QpidRestTestCase.waitForAttributeChanged(QpidRestTestCase.java:117)
	at org.apache.qpid.server.store.berkeleydb.replication.BDBHAVirtualHostNodeRestTest.testIntruderProtection(BDBHAVirtualHostNodeRestTest.java:311)

The log analysis showed that the issue occurs in the following scenario:

• 2-node cluster is created
• intruder node is connected
• node1 is shutdown by intruder protection
• node2 intruder protection is triggered and task to close VHN children is scheduled in Broker configuration thread. At the same time STATE event is issued by JE on transition from REPLICA into UNKNOWN (as majority is lost). The state change logic is invoked in the ReplicatedEnvironmentFacade StateShange executor which in turns performs VH close in VHN configuration thread and blocks until VH close is completed.
• As result, VHN configuration thread will be performing VHN children close caused by intruder protection, StateChange executor thread will be waiting for completion of VH close task which is scheduled as a separate task, Broker configuration thread will be performing REF.close waiting for shutdown of StateChange executor. When task to close VHN children is complete is schedules task in broker configuration thread to close configuration store. The latter can only be performed after intruder protection logic is completed.
• Thus, we have an effective dead lock, when tasks block each other threads.

It seems that REF.close in intruder protection functionality is not only redundant but harmful as it causes the effective dead lock. The deadlock resolves by timeout on waiting for a task executor shutdown.