Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-4107

OOME while writing WAL checksum causes corrupt WAL

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.90.1
    • Fix Version/s: None
    • Component/s: regionserver, wal
    • Labels:
      None
    • Environment:

      CentOS 5.5x64

      Description

      An issue was observed where upon shutdown of a regionserver the regionserver log was corrupt. It appears from the following stacktrace that an Java heap memory exception occurred while writing the checksum to the WAL. Corrupting the WAL can potentially cause data loss.

      2011-07-14 14:54:53,741 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog
      java.io.IOException: Reflection
      at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
      at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:987)
      at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:964)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.GeneratedMethodAccessor1336.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
      ... 2 more
      Caused by: java.lang.OutOfMemoryError: Java heap space
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$Packet.<init>(DFSClient.java:2375)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:3271)
      at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
      at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3354)
      at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
      at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
      ... 6 more

        Attachments

        1. regionserver.oom.log
          4 kB
          Dave Latham
        2. master.splitting.log
          12 kB
          Dave Latham

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              asautins Andy Sautins
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: