[ACCUMULO-3750] Bad instance.secret causes master to repeatedly fail fast attempting to acquire lock

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.6.2
• Fix Version/s: 1.6.3, 1.7.0
• Component/s: master
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 24/Apr/15 18:49
• Updated: 27/Apr/15 21:59
• Resolved: 27/Apr/15 21:59

Description

Accidentally restarted a small cluster with bad configuration (missing instance.secret). The tabletservers bailed out quickly, but the master sat in a tight loop trying to get the lock.

2015-04-23 11:48:12,356 [trace.DistributedTrace] INFO : SpanReceiver org.apache.accumulo.tracer.ZooTraceClient was loaded successfully.
2015-04-23 11:48:12,357 [master.Master] INFO : trying to get master lock
2015-04-23 11:48:12,395 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:13,043 [server.Accumulo] WARN : System swappiness setting is greater than ten (60) which can cause time-sensitive operations to be delayed.  Accumulo is time sensitive because it needs to maintain distributed lo
ck agreement.
2015-04-23 11:48:13,410 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:14,418 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:15,426 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:16,433 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:17,440 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:18,449 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-

Looks like the only case which exits the Master when the lock is failed to be acquired is an illegal state where the master thinks it already has the lock.

If we get a NoAuthException, we should not attempt to get the lock again.

Issue Links

• is related to: ACCUMULO-3752 Provide means to inject Halt implementation