[ACCUMULO-4038] recovery failure (no node for WAL)

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.8.0
• Component/s: master
• Labels: 18_qa_bug

Environment

20 node AWS cluster

• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 23/Oct/15 18:20
• Updated: 23/Oct/15 19:40
• Resolved: 23/Oct/15 19:24

Description

I was testing the new agitator, and system failed to make progress after this error in the master log:

2015-10-23 18:17:41,212 [master.Master] ERROR: Error processing table state for store Normal Tablets
org.apache.accumulo.server.log.WalStateManager$WalMarkerException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /accumulo/046bd887-d524-4fcb-b70f-3bdbdbcb3bab/wals/worker06:9997[35095adcb800018]
        at org.apache.accumulo.server.log.WalStateManager.getWalsInUse(WalStateManager.java:150)
        at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:253)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /accumulo/046bd887-d524-4fcb-b70f-3bdbdbcb3bab/wals/worker06:9997[35095adcb800018]
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
        at org.apache.accumulo.fate.zookeeper.ZooReader.getChildren(ZooReader.java:151)
        at org.apache.accumulo.server.log.WalStateManager.getWalsInUse(WalStateManager.java:143)
        ... 1 more