Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-2461

Split doesn't handle IOExceptions when creating new region reference files

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      I was testing an HDFS patch which had a bug in it, so it happened to throw an NPE during a split with the following trace:

      2010-04-16 19:18:20,727 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region TestTable,-1945465867<1271449232310>,1271453785648
      java.lang.NullPointerException
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.enqueueCurrentPacket(DFSClient.java:3124)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3220)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3306)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3255)
      at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
      at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
      at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:560)
      at org.apache.hadoop.hbase.util.FSUtils.create(FSUtils.java:95)
      at org.apache.hadoop.hbase.io.Reference.write(Reference.java:129)
      at org.apache.hadoop.hbase.regionserver.StoreFile.split(StoreFile.java:498)
      at org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:682)
      at org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:162)
      at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:95)

      After that, my region was gone, any further writes to it would fail.

      Attachments

        1. 2461.txt
          8 kB
          Michael Stack
        2. 2461-v10.txt
          71 kB
          Michael Stack
        3. 2461-v2.txt
          22 kB
          Michael Stack
        4. 2461-v3.txt
          36 kB
          Michael Stack
        5. 2461-v4.txt
          43 kB
          Michael Stack
        6. 2461-v6.txt
          53 kB
          Michael Stack
        7. 2461-v7.txt
          70 kB
          Michael Stack
        8. 2461-v8.txt
          70 kB
          Michael Stack
        9. ugly_but_might_work.txt
          2 kB
          Michael Stack

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stack Michael Stack
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment