[AMQ-5549] Shared Filesystem Master/Slave using NFSv4 allows both brokers become active at the same time

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 5.10.1
• Fix Version/s: 5.12.0
• Component/s: Broker, Message Store

Environment

• CentOS Linux 6
• OpenJDK 1.7
• ActiveMQ 5.10.1

• Assignee: Gary Tully
• Reporter: Heikki Manninen
• Created: 29/Jan/15 07:13
• Updated: 03/Jul/15 11:48
• Resolved: 03/Jul/15 11:47

Description

Identical ActiveMQ master and slave brokers are installed on CentOS Linux 6 virtual machines. There is a third virtual machine (also CentOS 6) providing an NFSv4 share for the brokers KahaDB.

Both brokers are started and the master broker acquires file lock on the lock file and the slave broker sits in a loop and waits for a lock as expected. Also changing brokers work as expected.

Once the network connection of the NFS server is disconnected both master and slave NFS mounts block and slave broker stops logging file lock re-tries. After a short while after bringing the network connection back the mounts come back and the slave broker is able to acquire the lock simultaneously. Both brokers accept client connections.

In this situation it is also possible to stop and start both individual brokers many times and they are always able to acquire the lock even if the other one is already running. Only after stopping both brokers and starting them again is the situation back to normal.

• NFS server:
  • CentOS Linux 6
  • NFS v4 export options: rw,sync
  • NFS v4 grace time 45 seconds
  • NFS v4 lease time 10 seconds

• NFS client:
  • CentOS Linux 6
  • NFS mount options: nfsvers=4,proto=tcp,hard,wsize=65536,rsize=65536

• ActiveMQ configuration (otherwise default):

        <persistenceAdapter>
            <kahaDB directory="${activemq.data}/kahadb">
              <locker>
                <shared-file-locker lockAcquireSleepInterval="1000"/>
              </locker>
            </kahaDB>
        </persistenceAdapter>

Issue Links

• depends upon: AMQ-4705 Add keep alive support to shared file locker
• is related to: AMQ-5568 Deleting lock file on broker shut down can take a master broker down