Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      MacOSX, Hadoop 1.0.3, zookeeper 3.3.3

      Description

      I shut my machine down with Accumulo, Zookeeper, and HDFS running. When I restarted it, Accumulo failed to recover its write ahead log because it was zero length. I wondered if this was because I shutdown HDFS so I tried the following on my single node Accumulo instance.

      • start HDFS and zookeeper
      • init & start Accumulo
      • created a table and insert some data
      • pkill -f java
      • restart everything
      • Accumulo fails to start because walog is zero length

      Saw excpetions like the following

      06 18:58:44,581 [log.SortedLogRecovery] INFO : Looking at mutations from /accumulo/recovery/def72721-5c64-4755-87cc-2e8cfc3002b7 for !0;!0<<
      06 18:58:44,590 [tabletserver.TabletServer] WARN : exception trying to assign tablet !0;!0<< /root_tablet
      java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: Unable to read log entries
              at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1458)
              at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1295)
              at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1134)
              at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1121)
              at org.apache.accumulo.server.tabletserver.TabletServer$AssignmentHandler.run(TabletServer.java:2477)
              at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
              at java.lang.Thread.run(Thread.java:680)
      Caused by: java.io.IOException: java.lang.RuntimeException: Unable to read log entries
              at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.recover(TabletServerLogger.java:428)
              at org.apache.accumulo.server.tabletserver.TabletServer.recover(TabletServer.java:3206)
              at org.apache.accumulo.server.tabletserver.Tablet.<init>(Tablet.java:1426)
              ... 6 more
      Caused by: java.lang.RuntimeException: Unable to read log entries
              at org.apache.accumulo.server.tabletserver.log.SortedLogRecovery.findLastStartToFinish(SortedLogRecovery.java:125)
              at org.apache.accumulo.server.tabletserver.log.SortedLogRecovery.recover(SortedLogRecovery.java:89)
              at org.apache.accumulo.server.tabletserver.log.TabletServerLogger.recover(TabletServerLogger.java:426)
              ... 8 more
      
      

      When trying to run LogReader on the files, it prints nothing.

      $ ./bin/accumulo org.apache.accumulo.server.logger.LogReader /accumulo/recovery/def72721-5c64-4755-87cc-2e8cfc3002b7
      06 19:04:37,147 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      $ ./bin/accumulo org.apache.accumulo.server.logger.LogReader /accumulo/wal/127.0.0.1+40200/def72721-5c64-4755-87cc-2e8cfc3002b7
      $ 
      

        Activity

        Keith Turner made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Assignee Eric Newton [ ecn ] Keith Turner [ kturner ]
        Resolution Fixed [ 1 ]
        Keith Turner created issue -

          People

          • Assignee:
            Keith Turner
            Reporter:
            Keith Turner
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development