HBase
  1. HBase
  2. HBASE-7513

HDFSBlocksDistribution shouldn't send NPEs when something goes wrong

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.94.4, 0.95.2
    • Fix Version/s: 0.94.5, 0.95.0
    • Component/s: None
    • Labels:
      None

      Description

      I saw a pretty weird failure on a cluster with corrupted files and this particular exception really threw me off:

      2013-01-07 09:58:59,054 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=redacted., starting to roll back the global memstore size.
      java.io.IOException: java.io.IOException: java.lang.NullPointerException: empty hosts
      	at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:548)
      	at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:461)
      	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3814)
      	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3762)
      	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
      	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
      	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.io.IOException: java.lang.NullPointerException: empty hosts
      	at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:403)
      	at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:256)
      	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2995)
      	at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:523)
      	at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:521)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	... 3 more
      Caused by: java.lang.NullPointerException: empty hosts
      	at org.apache.hadoop.hbase.HDFSBlocksDistribution.addHostsAndBlockWeight(HDFSBlocksDistribution.java:123)
      	at org.apache.hadoop.hbase.util.FSUtils.computeHDFSBlocksDistribution(FSUtils.java:597)
      	at org.apache.hadoop.hbase.regionserver.StoreFile.computeHDFSBlockDistribution(StoreFile.java:492)
      	at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:521)
      	at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:602)
      	at org.apache.hadoop.hbase.regionserver.Store$1.call(Store.java:380)
      	at org.apache.hadoop.hbase.regionserver.Store$1.call(Store.java:375)
      	... 8 more
      2013-01-07 09:58:59,059 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of region "redacted" failed, marking as FAILED_OPEN in ZK
      

      This is what the code looks like:

      if (hosts == null || hosts.length == 0) {
       throw new NullPointerException("empty hosts");
      }
      

      So hosts can exist but we send an NPE anyways? And then this is wrapped in Store by:

      } catch (ExecutionException e) {
        throw new IOException(e.getCause());
      

      FWIW there's another NPE thrown in HDFSBlocksDistribution.addHostAndBlockWeight and it looks wrong.

      We should change the code to just skip computing the locality if it's missing and not throw big ugly exceptions. In this case the region would fail opening later anyways but at least the error message will be clearer.

      1. HBASE-7513-0.patch
        2 kB
        Elliott Clark
      2. HBASE-7513-094-1.patch
        5 kB
        Elliott Clark
      3. HBASE-7513-1.patch
        5 kB
        Elliott Clark

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #11 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/11/)
        HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430580)

        Result = FAILURE
        eclark :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #11 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/11/ ) HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430580) Result = FAILURE eclark : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #95 (See https://builds.apache.org/job/HBase-0.94-security/95/)
        HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430580)

        Result = SUCCESS
        eclark :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #95 (See https://builds.apache.org/job/HBase-0.94-security/95/ ) HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430580) Result = SUCCESS eclark : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #336 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/336/)
        HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430560)

        Result = FAILURE
        eclark :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #336 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/336/ ) HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430560) Result = FAILURE eclark : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #715 (See https://builds.apache.org/job/HBase-0.94/715/)
        HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430580)

        Result = FAILURE
        eclark :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #715 (See https://builds.apache.org/job/HBase-0.94/715/ ) HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430580) Result = FAILURE eclark : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3711 (See https://builds.apache.org/job/HBase-TRUNK/3711/)
        HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430560)

        Result = FAILURE
        eclark :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3711 (See https://builds.apache.org/job/HBase-TRUNK/3711/ ) HBASE-7513 HDFSBlocksDistribution shouldn't send NPEs when something goes wrong (Revision 1430560) Result = FAILURE eclark : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHDFSBlocksDistribution.java
        Hide
        Elliott Clark added a comment -

        Committed to both trunk and 0.94.

        Thanks JD.

        Show
        Elliott Clark added a comment - Committed to both trunk and 0.94. Thanks JD.
        Hide
        Elliott Clark added a comment -

        0.94 version

        Show
        Elliott Clark added a comment - 0.94 version
        Hide
        Elliott Clark added a comment -

        Trunk version was Committed revision 1430560. I'll get a 0.94 version up soon.

        Show
        Elliott Clark added a comment - Trunk version was Committed revision 1430560. I'll get a 0.94 version up soon.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12563805/HBASE-7513-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.TestLocalHBaseCluster
        org.apache.hadoop.hbase.replication.TestReplication

        -1 core zombie tests. There are 1 zombie test(s):

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563805/HBASE-7513-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.TestLocalHBaseCluster org.apache.hadoop.hbase.replication.TestReplication -1 core zombie tests . There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3933//console This message is automatically generated.
        Hide
        Jean-Daniel Cryans added a comment -

        +1

        Show
        Jean-Daniel Cryans added a comment - +1
        Hide
        Elliott Clark added a comment -
        • Shortened the in code comments.
        • Moved the comment into the javadoc.
        • Added a test for this and other uses of HDFSBlocksDistribution.
        Show
        Elliott Clark added a comment - Shortened the in code comments. Moved the comment into the javadoc. Added a test for this and other uses of HDFSBlocksDistribution.
        Hide
        Elliott Clark added a comment -

        Sure.

        Show
        Elliott Clark added a comment - Sure.
        Hide
        Jean-Daniel Cryans added a comment -

        Not a fan of the copy-paste, maybe extract into the class javadoc?

        Show
        Jean-Daniel Cryans added a comment - Not a fan of the copy-paste, maybe extract into the class javadoc?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12563656/HBASE-7513-0.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor

        -1 core zombie tests. There are 6 zombie test(s):

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563656/HBASE-7513-0.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor -1 core zombie tests . There are 6 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3918//console This message is automatically generated.
        Hide
        Elliott Clark added a comment -

        Pretty easy patch.

        Since the hdfs block locality stuff isn't really used for anything other than metrics, just ignoring the case where no hosts are passed in has no adverse effects.

        Show
        Elliott Clark added a comment - Pretty easy patch. Since the hdfs block locality stuff isn't really used for anything other than metrics, just ignoring the case where no hosts are passed in has no adverse effects.
        Hide
        Elliott Clark added a comment -

        +1 I'll get this. Creating an HDFS Block Distribution shouldn't fail opening a store file.

        Show
        Elliott Clark added a comment - +1 I'll get this. Creating an HDFS Block Distribution shouldn't fail opening a store file.

          People

          • Assignee:
            Elliott Clark
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development