HBase
  1. HBase
  2. HBASE-5282

Possible file handle leak with truncated HLog file.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.5, 0.92.0, 0.94.0
    • Fix Version/s: 0.92.1, 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When debugging hbck, found that the code responsible for this exception can leak open file handles.

      12/01/15 05:58:11 INFO regionserver.HRegion: Replaying edits from hdfs://haus01.
      sf.cloudera.com:56020/hbase-jon/test5/98a1e7255731aae44b3836641840113e/recovered
      .edits/0000000000003211315; minSequenceid=3214658
      12/01/15 05:58:11 ERROR handler.OpenRegionHandler: Failed open of region=test5,8
      \x90\x00\x00\x00\x00\x00\x00/000005_0,1326597390073.98a1e7255731aae44b3836641840
      113e.
      java.io.EOFException
              at java.io.DataInputStream.readByte(DataInputStream.java:250)
              at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:299)
              at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:320)
              at org.apache.hadoop.io.Text.readString(Text.java:400)
              at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1486)
              at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1437)
              at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
              at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
              at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:57)
              at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158)
              at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:572)
              at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:1940)
              at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:1896)
              at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:366)
              at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2661)
              at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2647)
              at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:312)
              at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:99)
              at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:158)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:619)
      
      1. hbase-5282.v2.patch
        1 kB
        Jonathan Hsieh
      2. hbase-5282.patch
        1 kB
        Jonathan Hsieh

        Activity

        Hide
        Jonathan Hsieh added a comment -

        When debugging, open region file was attempting to open either a truncated or 0 size hlogfile (which is throws IOException at out from getReader), and leaking a handle on every open attempt.

        Patch applies on 0.92 and trunk.

        Show
        Jonathan Hsieh added a comment - When debugging, open region file was attempting to open either a truncated or 0 size hlogfile (which is throws IOException at out from getReader), and leaking a handle on every open attempt. Patch applies on 0.92 and trunk.
        Hide
        Ted Yu added a comment -

        reader.close() may throw IOE.
        I think we should protect the execution of status.cleanup().

        Show
        Ted Yu added a comment - reader.close() may throw IOE. I think we should protect the execution of status.cleanup().
        Hide
        Jonathan Hsieh added a comment -

        True, but #replayRecoveredEdits is only used in one place, wrapped with try catch that checks for IOE and seems like reasonable behavior:

        #replayRecoverededitsIfAny(...)

              try {
                seqid = replayRecoveredEdits(edits, seqid, reporter);
              } catch (IOException e) {
                boolean skipErrors = conf.getBoolean("hbase.skip.errors", false);
                if (skipErrors) {
                  Path p = HLog.moveAsideBadEditsFile(fs, edits);
                  LOG.error("hbase.skip.errors=true so continuing. Renamed " + edits +
                    " as " + p, e);
                } else {
                  throw e;
                }
              }
        

        What do you mean by protect status.cleanup()? Check for status == null? (it cannot be).

        Show
        Jonathan Hsieh added a comment - True, but #replayRecoveredEdits is only used in one place, wrapped with try catch that checks for IOE and seems like reasonable behavior: #replayRecoverededitsIfAny(...) try { seqid = replayRecoveredEdits(edits, seqid, reporter); } catch (IOException e) { boolean skipErrors = conf.getBoolean( "hbase.skip.errors" , false ); if (skipErrors) { Path p = HLog.moveAsideBadEditsFile(fs, edits); LOG.error( "hbase.skip.errors= true so continuing. Renamed " + edits + " as " + p, e); } else { throw e; } } What do you mean by protect status.cleanup()? Check for status == null ? (it cannot be).
        Hide
        Ted Yu added a comment -

        What I meant is that if close() throws IOE, status.cleanup() would be skipped.
        status.cleanup() can be placed before the call to close().

        Show
        Ted Yu added a comment - What I meant is that if close() throws IOE, status.cleanup() would be skipped. status.cleanup() can be placed before the call to close().
        Hide
        Jonathan Hsieh added a comment -

        Ah, got it. Good catch.

        Show
        Jonathan Hsieh added a comment - Ah, got it. Good catch.
        Hide
        Jonathan Hsieh added a comment -

        Updated to call status.cleanup() before close.

        Show
        Jonathan Hsieh added a comment - Updated to call status.cleanup() before close.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12511978/hbase-5282.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.hfile.TestHFileBlock
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/853//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/853//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/853//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511978/hbase-5282.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -140 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/853//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/853//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/853//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12511993/hbase-5282.v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.io.hfile.TestHFileBlock
        org.apache.hadoop.hbase.mapreduce.TestImportTsv

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/854//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/854//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/854//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511993/hbase-5282.v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -140 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/854//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/854//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/854//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        +1 on patch v2.

        Show
        Ted Yu added a comment - +1 on patch v2.
        Hide
        Jonathan Hsieh added a comment -

        I'll commit later today.

        Show
        Jonathan Hsieh added a comment - I'll commit later today.
        Hide
        Lars Hofhansl added a comment -

        +1 on v2

        Show
        Lars Hofhansl added a comment - +1 on v2
        Hide
        Jonathan Hsieh added a comment -

        First code commit! Thanks for the review Ted, Lars!

        Show
        Jonathan Hsieh added a comment - First code commit! Thanks for the review Ted, Lars!
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #265 (See https://builds.apache.org/job/HBase-0.92/265/)
        HBASE-5282 Possible file handle leak with truncated HLog file

        jmhsieh :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #265 (See https://builds.apache.org/job/HBase-0.92/265/ ) HBASE-5282 Possible file handle leak with truncated HLog file jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #92 (See https://builds.apache.org/job/HBase-TRUNK-security/92/)
        HBASE-5282 Possible file handle leak with truncated HLog file

        jmhsieh :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #92 (See https://builds.apache.org/job/HBase-TRUNK-security/92/ ) HBASE-5282 Possible file handle leak with truncated HLog file jmhsieh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2648 (See https://builds.apache.org/job/HBase-TRUNK/2648/)
        HBASE-5282 Possible file handle leak with truncated HLog file

        jmhsieh :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2648 (See https://builds.apache.org/job/HBase-TRUNK/2648/ ) HBASE-5282 Possible file handle leak with truncated HLog file jmhsieh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #90 (See https://builds.apache.org/job/HBase-0.92-security/90/)
        HBASE-5282 Possible file handle leak with truncated HLog file

        jmhsieh :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #90 (See https://builds.apache.org/job/HBase-0.92-security/90/ ) HBASE-5282 Possible file handle leak with truncated HLog file jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java

          People

          • Assignee:
            Jonathan Hsieh
            Reporter:
            Jonathan Hsieh
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development