HBase
  1. HBase
  2. HBASE-5975

Failed suppression of fs shutdown hook with Hadoop 2.0.0

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Unit test failed with error: Failed suppression of fs shutdown hook

      ShutdownHookManager.deleteShutdownHook failed to delete the hook since it should be runnable instead of a thread for HADOOP 2.0.0.
      For other HADOOP version, runnable should work too since thread implements runnable.

      1. 5975+5966.patch
        6 kB
        Jimmy Xiang
      2. hbase-5975.patch
        5 kB
        Jimmy Xiang

        Activity

        Hide
        Ted Yu added a comment -

        @Jimmy:
        Which unit test exhibited the failure ?

        Thanks

        Show
        Ted Yu added a comment - @Jimmy: Which unit test exhibited the failure ? Thanks
        Hide
        Jimmy Xiang added a comment -

        Many tests. I am testing with TestAcidGuarantees

        Show
        Jimmy Xiang added a comment - Many tests. I am testing with TestAcidGuarantees
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526205/hbase-5975.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.coprocessor.TestMasterObserver

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1821//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1821//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1821//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526205/hbase-5975.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterObserver Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1821//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1821//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1821//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        TestMasterObserver passed with the patch.

        TestAcidGuarantees passed with patch against hadoop 2.0

        +1 on patch.

        Show
        Ted Yu added a comment - TestMasterObserver passed with the patch. TestAcidGuarantees passed with patch against hadoop 2.0 +1 on patch.
        Hide
        Ted Yu added a comment -

        Will integrate the patch tomorrow morning if there is no objection.

        Show
        Ted Yu added a comment - Will integrate the patch tomorrow morning if there is no objection.
        Hide
        Andrew Purtell added a comment - - edited

        Ted, let's make an informal rule that we don't commit unless all tests pass locally with Hadoop 2.0.0, or the failures are known due to some other issue, until there is a Hudson project for it. So the JIRA comment on resolution should include the test summary.

        Show
        Andrew Purtell added a comment - - edited Ted, let's make an informal rule that we don't commit unless all tests pass locally with Hadoop 2.0.0, or the failures are known due to some other issue, until there is a Hudson project for it. So the JIRA comment on resolution should include the test summary.
        Hide
        Ted Yu added a comment -

        I applied patch from HBASE-5966.
        I saw this failure:

        testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport)  Time elapsed: 0.049 sec  <<< ERROR!
        java.lang.reflect.UndeclaredThrowableException
          at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
          at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:134)
          at org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:183)
        ...
        Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: Call From sea-lab-1/10.249.196.102 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188)
          at $Proxy92.getNewApplication(Unknown Source)
          at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:132)
          ... 41 more
        Caused by: java.net.ConnectException: Call From sea-lab-1/10.249.196.102 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
          at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:725)
          at org.apache.hadoop.ipc.Client.call(Client.java:1160)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:185)
          ... 43 more
        Caused by: java.net.ConnectException: Connection refused
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
        

        I think Jonathan H. is working on the above issue in HBASE-5876.

        Show
        Ted Yu added a comment - I applied patch from HBASE-5966 . I saw this failure: testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time elapsed: 0.049 sec <<< ERROR! java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135) at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:134) at org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:183) ... Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: Call From sea-lab-1/10.249.196.102 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http: //wiki.apache.org/hadoop/ConnectionRefused at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188) at $Proxy92.getNewApplication(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:132) ... 41 more Caused by: java.net.ConnectException: Call From sea-lab-1/10.249.196.102 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http: //wiki.apache.org/hadoop/ConnectionRefused at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:725) at org.apache.hadoop.ipc.Client.call(Client.java:1160) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:185) ... 43 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) I think Jonathan H. is working on the above issue in HBASE-5876 .
        Hide
        Ted Yu added a comment -

        Here is another failure:

        testMultiRegionTable(org.apache.hadoop.hbase.mapred.TestTableMapReduce)  Time elapsed: 12.749 sec  <<< ERROR!
        java.io.IOException
          at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
          at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
          at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:483)
          at org.apache.hadoop.mapreduce.Job$1.run(Job.java:322)
          at org.apache.hadoop.mapreduce.Job$1.run(Job.java:319)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
          at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:319)
          at org.apache.hadoop.mapreduce.Job.isSuccessful(Job.java:615)
          at org.apache.hadoop.mapred.JobClient$NetworkedJob.isSuccessful(JobClient.java:284)
          at org.apache.hadoop.hbase.mapred.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:156)
          at org.apache.hadoop.hbase.mapred.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:138)
        

        ClientServiceDelegate didn't provide detail for the IOE.

        Show
        Ted Yu added a comment - Here is another failure: testMultiRegionTable(org.apache.hadoop.hbase.mapred.TestTableMapReduce) Time elapsed: 12.749 sec <<< ERROR! java.io.IOException at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317) at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385) at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:483) at org.apache.hadoop.mapreduce.Job$1.run(Job.java:322) at org.apache.hadoop.mapreduce.Job$1.run(Job.java:319) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:319) at org.apache.hadoop.mapreduce.Job.isSuccessful(Job.java:615) at org.apache.hadoop.mapred.JobClient$NetworkedJob.isSuccessful(JobClient.java:284) at org.apache.hadoop.hbase.mapred.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:156) at org.apache.hadoop.hbase.mapred.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:138) ClientServiceDelegate didn't provide detail for the IOE.
        Hide
        Jimmy Xiang added a comment -

        @Ted, which patch from hbase-5966 did you use? You should use the patch I posted: hbase-5966.patch.
        Another thing, you need to make sure JAVA_HOME is set properly.

        I tried again and TestTableMapReduce is green for me:

        Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.037 sec
        Running org.apache.hadoop.hbase.mapreduce.TestImportExport
        Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 47.781 sec <<< FAILURE!
        Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.161 sec

        Here is the command I used:

        mvn -PlocalTests -Dhadoop.profile=23 -Dhadoop.version=2.0.0-SNAPSHOT -Dtest=TestTableMapReduce,TestImportExport clean test

        I will follow up with Jon on HBASE-5876.

        The combined patch I used is attached too.

        Show
        Jimmy Xiang added a comment - @Ted, which patch from hbase-5966 did you use? You should use the patch I posted: hbase-5966.patch. Another thing, you need to make sure JAVA_HOME is set properly. I tried again and TestTableMapReduce is green for me: Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.037 sec Running org.apache.hadoop.hbase.mapreduce.TestImportExport Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 47.781 sec <<< FAILURE! Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.161 sec Here is the command I used: mvn -PlocalTests -Dhadoop.profile=23 -Dhadoop.version=2.0.0-SNAPSHOT -Dtest=TestTableMapReduce,TestImportExport clean test I will follow up with Jon on HBASE-5876 . The combined patch I used is attached too.
        Hide
        Ted Yu added a comment -

        I tried the combined patch.
        TestTableMapReduce passed.

        Do you see other failed test(s) ?

        Show
        Ted Yu added a comment - I tried the combined patch. TestTableMapReduce passed. Do you see other failed test(s) ?
        Hide
        Jimmy Xiang added a comment -

        TestMetaMigrationRemovingHTD and TestLogRolling failed too. I filed HBASE-5985 and HBASE-5984 for them.

        Show
        Jimmy Xiang added a comment - TestMetaMigrationRemovingHTD and TestLogRolling failed too. I filed HBASE-5985 and HBASE-5984 for them.
        Hide
        Ted Yu added a comment -

        @Andy:
        Are you fine with tracking the known test failures in their respective JIRA ?

        Show
        Ted Yu added a comment - @Andy: Are you fine with tracking the known test failures in their respective JIRA ?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526371/5975%2B5966.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestAtomicOperation
        org.apache.hadoop.hbase.TestDrainingServer

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1833//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1833//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1833//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526371/5975%2B5966.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.TestDrainingServer Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1833//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1833//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1833//console This message is automatically generated.
        Hide
        Andrew Purtell added a comment -

        Are you fine with tracking the known test failures in their respective JIRA ?

        +1

        Show
        Andrew Purtell added a comment - Are you fine with tracking the known test failures in their respective JIRA ? +1
        Hide
        Ted Yu added a comment -

        Integrated to 0.94 and trunk.

        Thanks for the patch Jimmy.

        Thanks for the review, Andy.

        Show
        Ted Yu added a comment - Integrated to 0.94 and trunk. Thanks for the patch Jimmy. Thanks for the review, Andy.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2861 (See https://builds.apache.org/job/HBase-TRUNK/2861/)
        HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy Xiang) (Revision 1336875)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2861 (See https://builds.apache.org/job/HBase-TRUNK/2861/ ) HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy Xiang) (Revision 1336875) Result = FAILURE tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #187 (See https://builds.apache.org/job/HBase-0.94/187/)
        HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy Xiang) (Revision 1336876)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #187 (See https://builds.apache.org/job/HBase-0.94/187/ ) HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy Xiang) (Revision 1336876) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #198 (See https://builds.apache.org/job/HBase-TRUNK-security/198/)
        HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy Xiang) (Revision 1336875)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #198 (See https://builds.apache.org/job/HBase-TRUNK-security/198/ ) HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy Xiang) (Revision 1336875) Result = FAILURE tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java
        Hide
        stack added a comment -

        Resolving. Ted applied this to 0.94 and trunk.

        Show
        stack added a comment - Resolving. Ted applied this to 0.94 and trunk.
        Hide
        stack added a comment -

        What you all want me to set up in jenkins? A build against 2.0.0? Should I put it in place of the 0.23 build we currently have?

        Show
        stack added a comment - What you all want me to set up in jenkins? A build against 2.0.0? Should I put it in place of the 0.23 build we currently have?
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #27 (See https://builds.apache.org/job/HBase-0.94-security/27/)
        HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy Xiang) (Revision 1336876)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #27 (See https://builds.apache.org/job/HBase-0.94-security/27/ ) HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy Xiang) (Revision 1336876) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java

          People

          • Assignee:
            Jimmy Xiang
            Reporter:
            Jimmy Xiang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development