Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-3750

Bad instance.secret causes master to repeatedly fail fast attempting to acquire lock

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.6.2
    • Fix Version/s: 1.6.3, 1.7.0
    • Component/s: master
    • Labels:
      None

      Description

      Accidentally restarted a small cluster with bad configuration (missing instance.secret). The tabletservers bailed out quickly, but the master sat in a tight loop trying to get the lock.

      2015-04-23 11:48:12,356 [trace.DistributedTrace] INFO : SpanReceiver org.apache.accumulo.tracer.ZooTraceClient was loaded successfully.
      2015-04-23 11:48:12,357 [master.Master] INFO : trying to get master lock
      2015-04-23 11:48:12,395 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
      2015-04-23 11:48:13,043 [server.Accumulo] WARN : System swappiness setting is greater than ten (60) which can cause time-sensitive operations to be delayed.  Accumulo is time sensitive because it needs to maintain distributed lo
      ck agreement.
      2015-04-23 11:48:13,410 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
      2015-04-23 11:48:14,418 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
      2015-04-23 11:48:15,426 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
      2015-04-23 11:48:16,433 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
      2015-04-23 11:48:17,440 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
      2015-04-23 11:48:18,449 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
      

      Looks like the only case which exits the Master when the lock is failed to be acquired is an illegal state where the master thinks it already has the lock.

      If we get a NoAuthException, we should not attempt to get the lock again.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                elserj Josh Elser
                Reporter:
                elserj Josh Elser
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h