Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
-
None
-
None
-
MacOSX, Hadoop 1.0.3, zookeeper 3.3.3
Description
I shut my machine down with Accumulo, Zookeeper, and HDFS running. When I restarted it, Accumulo failed to recover its write ahead log because it was zero length. I wondered if this was because I shutdown HDFS so I tried the following on my single node Accumulo instance.
- start HDFS and zookeeper
- init & start Accumulo
- created a table and insert some data
- pkill -f java
- restart everything
- Accumulo fails to start because walog is zero length
Saw excpetions like the following
06 18:58:44,581 [log.SortedLogRecovery] INFO : Looking at mutations from /accumulo/recovery/def72721-5c64-4755-87cc-2e8cfc3002b7 for !0;!0<< 06 18:58:44,590 [tabletserver.TabletServer] WARN : exception trying to assign tablet !0;!0<< /root_tablet java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: Unable to read log entries at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1458) at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1295) at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1134) at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1121) at org.apache.accumulo.server.tabletserver.TabletServer$AssignmentHandler.run(TabletServer.java:2477) at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) at java.lang.Thread.run(Thread.java:680) Caused by: java.io.IOException: java.lang.RuntimeException: Unable to read log entries at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.recover(TabletServerLogger.java:428) at org.apache.accumulo.server.tabletserver.TabletServer.recover(TabletServer.java:3206) at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1426) ... 6 more Caused by: java.lang.RuntimeException: Unable to read log entries at org.apache.accumulo.server.tabletserver.log.SortedLogRecovery.findLastStartToFinish(SortedLogRecovery.java:125) at org.apache.accumulo.server.tabletserver.log.SortedLogRecovery.recover(SortedLogRecovery.java:89) at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.recover(TabletServerLogger.java:426) ... 8 more
When trying to run LogReader on the files, it prints nothing.
$ ./bin/accumulo org.apache.accumulo.server.logger.LogReader /accumulo/recovery/def72721-5c64-4755-87cc-2e8cfc3002b7 06 19:04:37,147 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable $ ./bin/accumulo org.apache.accumulo.server.logger.LogReader /accumulo/wal/127.0.0.1+40200/def72721-5c64-4755-87cc-2e8cfc3002b7 $