Accumulo
  1. Accumulo
  2. ACCUMULO-1277

Race condition between master and tserver when acquiring tserver lock

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.4, 1.5.0
    • Component/s: master, tserver
    • Labels:
      None

      Description

      When restarting a stopped tserver, the following happens:
      The tserver (in TabletServer.announceExistence()) creates an entry in zookeeper at /accumulo/instance-id/tserver/host:port.

      This in turn triggers master to execute the call chain:
      LiveTServerSet.process(WatchedEvent)
      LiveTServerSet.scanServers()
      LiveTServerSet.checkServer(Set<TServerInstance>, Set<TServerInstance>, String, String)

      The checkServer() method checks to see if the ZooLock data has been created yet (if tserver loses the race, it has not yet been created) causing master to then delete the tserver node.

      When the tserver attempts to create the ZooLock, the parent path no longer exists and creating the lock fails. Eventually the tserver will time out waiting to create the lock, and fail to start.

      This problem is easier to reproduce in a smallish cluster using a single zookeeper node, where there is more latency between the tserver and zookeeper than there is between the master and zookeeper.

      This behavior was introduced in the fix for ACCUMULO-1049.

        Activity

        Hide
        Keith Turner added a comment -

        I had some code in place that delayed deleting empty tserver nodes, but it looks like I just dropped it. Ooppss. I'll take a look at this. Nice write up.

        Show
        Keith Turner added a comment - I had some code in place that delayed deleting empty tserver nodes, but it looks like I just dropped it. Ooppss. I'll take a look at this. Nice write up.
        Hide
        Daniel P Truitt added a comment -

        It seems like a lot of the locking complexity in Accumulo might be alleviated by upgrading to the 3.4.x versions of zookeeper, which supports multi-operation transactions.

        Show
        Daniel P Truitt added a comment - It seems like a lot of the locking complexity in Accumulo might be alleviated by upgrading to the 3.4.x versions of zookeeper, which supports multi-operation transactions.
        Hide
        Keith Turner added a comment -

        It seems like a lot of the locking complexity in Accumulo might be alleviated by upgrading to the 3.4.x versions of zookeeper, which supports multi-operation transactions.

        I have wanted to use that feature in the past. Maybe we can bump the version for 1.6 I think the zookeeper team is now recommending 3.4.5.

        Show
        Keith Turner added a comment - It seems like a lot of the locking complexity in Accumulo might be alleviated by upgrading to the 3.4.x versions of zookeeper, which supports multi-operation transactions. I have wanted to use that feature in the past. Maybe we can bump the version for 1.6 I think the zookeeper team is now recommending 3.4.5.
        Hide
        Hudson added a comment -

        Integrated in Accumulo-1.4.x #293 (See https://builds.apache.org/job/Accumulo-1.4.x/293/)
        ACCUMULO-1277 made master delay deleting lockless tserver nodes in zookeeper (Revision 1468589)

        Result = SUCCESS
        kturner :
        Files :

        • /accumulo/branches/1.4
        • /accumulo/branches/1.4/src
        • /accumulo/branches/1.4/src/core
        • /accumulo/branches/1.4/src/server
        • /accumulo/branches/1.4/src/server/src
        • /accumulo/branches/1.4/src/server/src/main/java/org/apache/accumulo/server/master/LiveTServerSet.java
        • /accumulo/branches/1.4/src/server/src/main/java/org/apache/accumulo/server/zookeeper/ZooLock.java
        Show
        Hudson added a comment - Integrated in Accumulo-1.4.x #293 (See https://builds.apache.org/job/Accumulo-1.4.x/293/ ) ACCUMULO-1277 made master delay deleting lockless tserver nodes in zookeeper (Revision 1468589) Result = SUCCESS kturner : Files : /accumulo/branches/1.4 /accumulo/branches/1.4/src /accumulo/branches/1.4/src/core /accumulo/branches/1.4/src/server /accumulo/branches/1.4/src/server/src /accumulo/branches/1.4/src/server/src/main/java/org/apache/accumulo/server/master/LiveTServerSet.java /accumulo/branches/1.4/src/server/src/main/java/org/apache/accumulo/server/zookeeper/ZooLock.java
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk #833 (See https://builds.apache.org/job/Accumulo-Trunk/833/)
        ACCUMULO-1277 made master delay deleting lockless tserver nodes in zookeeper (Revision 1468585)

        Result = SUCCESS
        kturner :
        Files :

        • /accumulo/trunk
        • /accumulo/trunk/assemble
        • /accumulo/trunk/core
        • /accumulo/trunk/examples
        • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java
        • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java
        • /accumulo/trunk/pom.xml
        • /accumulo/trunk/server
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/master/LiveTServerSet.java
        • /accumulo/trunk/src
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk #833 (See https://builds.apache.org/job/Accumulo-Trunk/833/ ) ACCUMULO-1277 made master delay deleting lockless tserver nodes in zookeeper (Revision 1468585) Result = SUCCESS kturner : Files : /accumulo/trunk /accumulo/trunk/assemble /accumulo/trunk/core /accumulo/trunk/examples /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java /accumulo/trunk/pom.xml /accumulo/trunk/server /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/master/LiveTServerSet.java /accumulo/trunk/src
        Hide
        Charles Ott added a comment - - edited

        The following stack trace may indicate the occurrence of this bug in your Accumulo 1.4.3 cluster when attempting to start tablet servers:

        Uncaught exception in TabletServer.main, exiting
                 java.lang.RuntimeException: java.lang.RuntimeException: Too many retries, exiting.
                         at org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(TabletServer.java:2684)
                         at org.apache.accumulo.server.tabletserver.TabletServer.run(TabletServer.java:2703)
                         at org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.java:3168)
                         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:597)
                         at org.apache.accumulo.start.Main$1.run(Main.java:89)
                         at java.lang.Thread.run(Thread.java:662)
                 Caused by: java.lang.RuntimeException: Too many retries, exiting.
                         at org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(TabletServer.java:2681)
                         ... 8 more
        
        Show
        Charles Ott added a comment - - edited The following stack trace may indicate the occurrence of this bug in your Accumulo 1.4.3 cluster when attempting to start tablet servers: Uncaught exception in TabletServer.main, exiting java.lang.RuntimeException: java.lang.RuntimeException: Too many retries, exiting. at org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(TabletServer.java:2684) at org.apache.accumulo.server.tabletserver.TabletServer.run(TabletServer.java:2703) at org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.java:3168) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.accumulo.start.Main$1.run(Main.java:89) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: Too many retries, exiting. at org.apache.accumulo.server.tabletserver.TabletServer.announceExistence(TabletServer.java:2681) ... 8 more

          People

          • Assignee:
            Keith Turner
            Reporter:
            Daniel P Truitt
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development