[ARTEMIS-2421] Implement periodic journal lock evaluation

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.4
• Fix Version/s: 2.11.0
• Component/s: Broker
• Assignee: Justin Bertram
• Reporter: Gaurav
• Created: 10/Jul/19 11:14
• Updated: 10/Jan/20 17:39
• Resolved: 03/Jan/20 03:03

Description

We have Live-Backup server configuration, single instance of Artemis Live server (2.6.4 version) backed up by single instance of Backup server.

Using shared file system as persistent storage.

Please refer attachments for both Live-Backup broker configuration.

Fail Over Scenario

1. Node 1 acting as Live node and serving requests whereas Node 2 acting as standby or passive node. No consumer is connected to these nodes
2. Pushed 5 messages and verify message count as 5
3. Perform NIC (Network) failure on Node 1 server ( i.e. Cluster is now unable to connect to Node 1) . This will make Node 2 as Active and we are also able to see previous 5 messages (pushed in step 2) successfully replicated on Node 2
4. Bring the network connection back for Node 1. This is where we are facing issues as now both nodes acting as Live nodes and getting continuous error as below:

AMQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur  if you have a backup node active at the same time as its live node. nodeID=cd323206-4adc-11e9-814b-506b8d4ee653

 

This situation bring entire cluster in inconsistent state and able to push messages on both the nodes.

Any pointer on this issue is much appreciated!

Issue Links

• is duplicated by: ARTEMIS-2250 Shared store lock is not monitored while running
• links to: GitHub Pull Request #2901