ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1546

"Unable to load database on disk" when restarting after node freeze

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.5
    • Fix Version/s: None
    • Component/s: server
    • Labels:
      None

      Description

      One of my zookeeper servers in a quorum of 3 froze (probably due to underlying hardware problems). When restarting, zookeeper fails to start with the following in zookeeper.log:

      2012-09-04 09:02:35,300 - INFO  [main:QuorumPeerConfig@90] - Reading configuration from: /etc/zookeeper/zoo.cfg
      2012-09-04 09:02:35,316 - INFO  [main:QuorumPeerConfig@310] - Defaulting to majority quorums
      2012-09-04 09:02:35,333 - INFO  [main:QuorumPeerMain@119] - Starting quorum peer
      2012-09-04 09:02:35,358 - INFO  [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
      2012-09-04 09:02:35,379 - INFO  [main:QuorumPeer@819] - tickTime set to 2000
      2012-09-04 09:02:35,380 - INFO  [main:QuorumPeer@830] - minSessionTimeout set to -1
      2012-09-04 09:02:35,380 - INFO  [main:QuorumPeer@841] - maxSessionTimeout set to -1
      2012-09-04 09:02:35,386 - INFO  [main:QuorumPeer@856] - initLimit set to 10
      2012-09-04 09:02:35,523 - INFO  [main:FileSnap@82] - Reading snapshot /var/zookeeper/version-2/snapshot.500017240
      2012-09-04 09:02:38,944 - ERROR [main:FileTxnSnapLog@226] - Failed to increment parent cversion for: /osp/production/scheduler/waitfordeps_tasks/per_period-3092724ef4d611e18411525400fff018-bulkload_histograms
      org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /osp/production/scheduler/waitfordeps_tasks/per_period-3092724ef4d611e18411525400fff018-bulkload_histograms
              at org.apache.zookeeper.server.DataTree.incrementCversion(DataTree.java:1218)
              at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:224)
              at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:152)
              at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
              at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76)
      2012-09-04 09:02:38,945 - FATAL [main:QuorumPeer@400] - Unable to load database on disk
      java.io.IOException: Failed to process transaction type: 2 error: KeeperErrorCode = NoNode for /osp/production/scheduler/waitfordeps_tasks/per_period-3092724ef4d611e18411525400fff018-bulkload_histograms
              at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:154)
              at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
              at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76)
      2012-09-04 09:02:38,946 - FATAL [main:QuorumPeerMain@87] - Unexpected exception, exiting abnormally
      java.lang.RuntimeException: Unable to run quorum server 
              at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:401)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103)
              at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76)
      Caused by: java.io.IOException: Failed to process transaction type: 2 error: KeeperErrorCode = NoNode for /osp/production/scheduler/waitfordeps_tasks/per_period-3092724ef4d611e18411525400fff018-bulkload_histograms
              at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:154)
              at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
              at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398)
              ... 3 more
      
      

      Removing data from /var/zookeeper/version-2 then restart seems to "fix" the problem (it gets a snapshot from one of the other nodes in the quorum).

      This is Zookeeper 3.3.5+19.5-1~squeeze-cdh3, i.e. from Cloudera's distribution.

        Activity

        Erik Forsberg created issue -
        Hide
        Alexander Sorokin added a comment -

        We had this issue completely take down our 5-node cluster last night. After digging around we found that the culprit. It's a snapshot that started with nodes /A/B/C present. Before it got to B, both B and C got removed. So, the snapshot contains only /A, but the txn log starts with "delete C". When it replays the log, it tries to increment the cversion of the parent, which is /A/B. It isn't present in the snapshot and the recovery crashes.

        This issue seems to have been fixed on trunk. I guess we'd just have to upgrade.

        Show
        Alexander Sorokin added a comment - We had this issue completely take down our 5-node cluster last night. After digging around we found that the culprit. It's a snapshot that started with nodes /A/B/C present. Before it got to B, both B and C got removed. So, the snapshot contains only /A, but the txn log starts with "delete C". When it replays the log, it tries to increment the cversion of the parent, which is /A/B. It isn't present in the snapshot and the recovery crashes. This issue seems to have been fixed on trunk. I guess we'd just have to upgrade.

          People

          • Assignee:
            Unassigned
            Reporter:
            Erik Forsberg
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development