HBase
  1. HBase
  2. HBASE-5966

MapReduce based tests broken on Hadoop 2.0.0-alpha

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0, 0.95.2
    • Fix Version/s: 0.94.1, 0.95.1
    • Component/s: mapreduce, test
    • Labels:
      None
    • Environment:

      Hadoop 2.0.0-alpha-SNAPSHOT, HBase 0.94.0-SNAPSHOT, Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)

    • Hadoop Flags:
      Reviewed

      Description

      Some fairly recent change in Hadoop 2.0.0-alpha has broken our MapReduce test rigging. Below is a representative error, can be easily reproduced with:

      mvn -PlocalTests -Psecurity \
        -Dhadoop.profile=23 -Dhadoop.version=2.0.0-SNAPSHOT \
        clean test \
        -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
      

      And the result:

      -------------------------------------------------------
       T E S T S
      -------------------------------------------------------
      Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.292 sec <<< FAILURE!
      
      -------------------------------------------------------------------------------
      Test set: org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
      -------------------------------------------------------------------------------
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.292 sec <<< FAILURE!
      testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce)  Time elapsed: 21.935 sec  <<< ERROR!
      java.lang.reflect.UndeclaredThrowableException
      	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
      	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:134)
      	at org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:183)
      	at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:216)
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:339)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:416)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
      	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244)
      	at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:151)
      	at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:129)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:616)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
      	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:616)
      	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
      	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
      	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
      	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
      Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: Call From acer.localdomain/192.168.122.1 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188)
      	at $Proxy89.getNewApplication(Unknown Source)
      	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:132)
      	... 45 more
      Caused by: java.net.ConnectException: Call From acer.localdomain/192.168.122.1 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
      	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:725)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1160)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:185)
      	... 47 more
      Caused by: java.net.ConnectException: Connection refused
      	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
      	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:522)
      	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:487)
      	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:469)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:563)
      	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:212)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1266)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1136)
      	... 48 more
      
      1. hbase-5966.patch
        2 kB
        Jimmy Xiang
      2. HBASE-5966.patch
        0.9 kB
        Devaraj K
      3. HBASE-5966-1.patch
        0.9 kB
        Devaraj K
      4. HBASE-5966-94.patch
        2 kB
        Gregory Chanan

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/)
        HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Gregory Chanan) (Revision 1363586)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/ ) HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Gregory Chanan) (Revision 1363586) Result = FAILURE jxiang : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #44 (See https://builds.apache.org/job/HBase-0.94-security/44/)
        HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Gregory Chanan) (Revision 1363586)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #44 (See https://builds.apache.org/job/HBase-0.94-security/44/ ) HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Gregory Chanan) (Revision 1363586) Result = FAILURE jxiang : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #344 (See https://builds.apache.org/job/HBase-0.94/344/)
        HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Gregory Chanan) (Revision 1363586)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #344 (See https://builds.apache.org/job/HBase-0.94/344/ ) HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Gregory Chanan) (Revision 1363586) Result = FAILURE jxiang : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Jimmy Xiang added a comment -

        Integrated to 0.94. Thank Greg for the patch, Lars for the review.

        Show
        Jimmy Xiang added a comment - Integrated to 0.94. Thank Greg for the patch, Lars for the review.
        Hide
        Lars Hofhansl added a comment -

        +1

        Show
        Lars Hofhansl added a comment - +1
        Hide
        Jimmy Xiang added a comment -

        looks good to me, will commit to 0.94 tonight if no objection.

        Show
        Jimmy Xiang added a comment - looks good to me, will commit to 0.94 tonight if no objection.
        Hide
        Gregory Chanan added a comment -

        Attached patch for 0.94. Ran TestTableMapReduce against both 1.0 and 2.0 hadoop profiles, both passed:

        mvn test -PlocalTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce

        -------------------------------------------------------
        T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 188.087 sec

        Results :

        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

        mvn test -PlocalTests -Dhadoop.profile=2.0 -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce

        -------------------------------------------------------
        T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 167.49 sec

        Results :

        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

        Show
        Gregory Chanan added a comment - Attached patch for 0.94. Ran TestTableMapReduce against both 1.0 and 2.0 hadoop profiles, both passed: mvn test -PlocalTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 188.087 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 mvn test -PlocalTests -Dhadoop.profile=2.0 -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 167.49 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
        Hide
        Lars Hofhansl added a comment -

        Discussed with Jimmy. Let's have this in 0.94.1

        Show
        Lars Hofhansl added a comment - Discussed with Jimmy. Let's have this in 0.94.1
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #2 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/2/)
        HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Revision 1337448)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #2 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/2/ ) HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Revision 1337448) Result = FAILURE stack : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2872 (See https://builds.apache.org/job/HBase-TRUNK/2872/)
        HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Revision 1337448)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2872 (See https://builds.apache.org/job/HBase-TRUNK/2872/ ) HBASE-5966 MapReduce based tests broken on Hadoop 2.0.0-alpha (Revision 1337448) Result = FAILURE stack : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        stack added a comment -

        It passed when I set JAVA_HOME. Thanks for the patch Jimmy.

        Show
        stack added a comment - It passed when I set JAVA_HOME. Thanks for the patch Jimmy.
        Hide
        Jimmy Xiang added a comment -

        Did you set JAVA_HOME?

        Show
        Jimmy Xiang added a comment - Did you set JAVA_HOME?
        Hide
        stack added a comment -

        What am I doing wrong? I tried it three times here on my laptop:

        -------------------------------------------------------------------------------
        Test set: org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        -------------------------------------------------------------------------------
        Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 64.363 sec <<< FAILURE!
        testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce)  Time elapsed: 30.455 sec  <<< FAILURE!
        java.lang.AssertionError
                at org.junit.Assert.fail(Assert.java:92)
                at org.junit.Assert.assertTrue(Assert.java:43)
                at org.junit.Assert.assertTrue(Assert.java:54)
                at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:151)
                at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:129)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
                at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
                at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
                at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
                at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
                at org.junit.rules.RunRules.evaluate(RunRules.java:18)
                at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
                at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
                at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
                at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
                at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
                at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
                at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
                at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
                at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
                at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
                at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
                at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
                at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
                at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
                at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
                at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
        

        I did this:

        $ MAVEN_OPTS="-Xmx2g" ~/bin/mvn/bin/mvn -PlocalTests -Psecurity   -Dhadoop.profile=23 -Dhadoop.version=2.0.0-SNAPSHOT   clean test   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        
        Show
        stack added a comment - What am I doing wrong? I tried it three times here on my laptop: ------------------------------------------------------------------------------- Test set: org.apache.hadoop.hbase.mapreduce.TestTableMapReduce ------------------------------------------------------------------------------- Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 64.363 sec <<< FAILURE! testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce) Time elapsed: 30.455 sec <<< FAILURE! java.lang.AssertionError at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:151) at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:129) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47) at org.junit.rules.RunRules.evaluate(RunRules.java:18) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) I did this: $ MAVEN_OPTS= "-Xmx2g" ~/bin/mvn/bin/mvn -PlocalTests -Psecurity -Dhadoop.profile=23 -Dhadoop.version=2.0.0-SNAPSHOT clean test -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        Hide
        Jimmy Xiang added a comment -

        Can we push this patch to trunk? Based on our testing, it does fix the MR test failures with HADOOP 2.0.0.

        Show
        Jimmy Xiang added a comment - Can we push this patch to trunk? Based on our testing, it does fix the MR test failures with HADOOP 2.0.0.
        Hide
        Jimmy Xiang added a comment -

        These tests are green on my box.

        Show
        Jimmy Xiang added a comment - These tests are green on my box.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526295/hbase-5966.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster
        org.apache.hadoop.hbase.coprocessor.TestClassLoading

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1827//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1827//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1827//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526295/hbase-5966.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster org.apache.hadoop.hbase.coprocessor.TestClassLoading Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1827//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1827//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1827//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        The patch is based on the patch Andrew posted.

        Together with the patch for HBASE-5975, TestTableMapReduce is green for me.

        Show
        Jimmy Xiang added a comment - The patch is based on the patch Andrew posted. Together with the patch for HBASE-5975 , TestTableMapReduce is green for me.
        Hide
        Andrew Purtell added a comment -

        I forgot to apply HBASE-5964 to the fresh checkout. After doing that, I see the same issue that Ted reports.

        Show
        Andrew Purtell added a comment - I forgot to apply HBASE-5964 to the fresh checkout. After doing that, I see the same issue that Ted reports.
        Hide
        Ted Yu added a comment -

        Combined with patches from HBASE-5964 and HBASE-5963, I saw this:

        testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce)  Time elapsed: 47.489 sec  <<< FAILURE!
        java.lang.AssertionError
          at org.junit.Assert.fail(Assert.java:92)
          at org.junit.Assert.assertTrue(Assert.java:43)
          at org.junit.Assert.assertTrue(Assert.java:54)
          at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:151)
          at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:129)
        

        In test output, I found:

        2012-05-09 10:05:11,141 ERROR [main] mapreduce.TableInputFormatBase(171): Cannot resolve the host name for /10.249.196.102 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name &apos;102.196.249.10.in-addr.arpa&apos;
        
        Show
        Ted Yu added a comment - Combined with patches from HBASE-5964 and HBASE-5963 , I saw this: testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce) Time elapsed: 47.489 sec <<< FAILURE! java.lang.AssertionError at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:151) at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:129) In test output, I found: 2012-05-09 10:05:11,141 ERROR [main] mapreduce.TableInputFormatBase(171): Cannot resolve the host name for /10.249.196.102 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name &apos;102.196.249.10.in-addr.arpa&apos;
        Hide
        Andrew Purtell added a comment -

        Above failure may have been due to locally applied patches for other ongoing issues with 2.0.0-alpha, so I ran it again with a clean 0.94 branch checkout and see this:

        -------------------------------------------------------------------------------
        Test set: org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        -------------------------------------------------------------------------------
        Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 205.635 sec <<< FAILURE!
        org.apache.hadoop.hbase.mapreduce.TestTableMapReduce  Time elapsed: 0 sec  <<< ERROR!
        java.io.IOException: Shutting down
        	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:203)
        	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:76)
        	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:632)
        	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:606)
        	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:554)
        	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:523)
        	at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.beforeClass(TestTableMapReduce.java:68)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:616)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
        	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
        	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
        	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:616)
        	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
        	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
        	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
        	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
        	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
        Caused by: java.lang.RuntimeException: Master not initialized after 200 seconds
        	at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:206)
        	at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:422)
        	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:196)
        	... 28 more
        

        Same result on a clean checkout of TRUNK.

        Show
        Andrew Purtell added a comment - Above failure may have been due to locally applied patches for other ongoing issues with 2.0.0-alpha, so I ran it again with a clean 0.94 branch checkout and see this: ------------------------------------------------------------------------------- Test set: org.apache.hadoop.hbase.mapreduce.TestTableMapReduce ------------------------------------------------------------------------------- Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 205.635 sec <<< FAILURE! org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Time elapsed: 0 sec <<< ERROR! java.io.IOException: Shutting down at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:203) at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:76) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:632) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:606) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:554) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:523) at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.beforeClass(TestTableMapReduce.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) Caused by: java.lang.RuntimeException: Master not initialized after 200 seconds at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:206) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:422) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:196) ... 28 more Same result on a clean checkout of TRUNK.
        Hide
        Jimmy Xiang added a comment -

        I saw lots of failures too. I am testing with patch for 5963 and 5966 now.

        Show
        Jimmy Xiang added a comment - I saw lots of failures too. I am testing with patch for 5963 and 5966 now.
        Hide
        Andrew Purtell added a comment -

        Testing with this patch ported to 0.94, via

        mvn -PlocalTests -Psecurity \
          -Dhadoop.profile=23 -Dhadoop.version=2.0.0-SNAPSHOT \
          clean test \
          -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        

        I see this failure:

        Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 45.473 sec <<< FAILURE!
        testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce)  Time elapsed: 11.876 sec  <<< ERROR!
        java.lang.NullPointerException
        	at org.apache.hadoop.net.DNS.reverseDns(DNS.java:93)
        	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.reverseDNS(TableInputFormatBase.java:200)
        	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:165)
        	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:451)
        	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:468)
        	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
        	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
        	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:416)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
        	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244)
        	at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:151)
        	at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:129)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:616)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
        	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
        	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
        	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
        	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
        	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
        	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
        	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
        	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
        	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
        	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:616)
        	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
        	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
        	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
        	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
        	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
        

        and probably there are more but this is the only one I've tried so far.

        Is it just me?

        Show
        Andrew Purtell added a comment - Testing with this patch ported to 0.94, via mvn -PlocalTests -Psecurity \ -Dhadoop.profile=23 -Dhadoop.version=2.0.0-SNAPSHOT \ clean test \ -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce I see this failure: Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 45.473 sec <<< FAILURE! testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce) Time elapsed: 11.876 sec <<< ERROR! java.lang.NullPointerException at org.apache.hadoop.net.DNS.reverseDns(DNS.java:93) at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.reverseDNS(TableInputFormatBase.java:200) at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:165) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:451) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:468) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244) at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:151) at org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:129) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47) at org.junit.rules.RunRules.evaluate(RunRules.java:18) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) and probably there are more but this is the only one I've tried so far. Is it just me?
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526135/HBASE-5966-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1814//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1814//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1814//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526135/HBASE-5966-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1814//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1814//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1814//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        I have updated the patch to fix the above failures.

        Show
        Devaraj K added a comment - I have updated the patch to fix the above failures.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526125/HBASE-5966.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        org.apache.hadoop.hbase.mapreduce.TestWALPlayer
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestImportExport
        org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
        org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
        org.apache.hadoop.hbase.mapreduce.TestImportTsv

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1813//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1813//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1813//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526125/HBASE-5966.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapreduce.TestWALPlayer org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportExport org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1813//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1813//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1813//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        I think this is coming after the changes of MAPREDUCE-3867.

        I have attached the patch to fix this.

        @Ashutosh: I don't think your failure is related to this because it impacts only the client to RM communication. As per your exception stack, RM is failing to start. Can you check the detailed log and environment for more details?

        Show
        Devaraj K added a comment - I think this is coming after the changes of MAPREDUCE-3867 . I have attached the patch to fix this. @Ashutosh: I don't think your failure is related to this because it impacts only the client to RM communication. As per your exception stack, RM is failing to start. Can you check the detailed log and environment for more details?
        Hide
        Ashutosh Jindal added a comment -

        This is the exception that i got while running testcases. Some other testcases are also failing with the same error.

         
        -------------------------------------------------------------------------------
        Test set: org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
        -------------------------------------------------------------------------------
        Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 177.132 sec <<< FAILURE!
        org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan  Time elapsed: 0 sec  <<< ERROR!
        org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.mapred.MiniMRCluster
        	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
        	at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:67)
        	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180)
        	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170)
        	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162)
        	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:154)
        	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:147)
        	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:140)
        	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:133)
        	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:128)
        	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1269)
        	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1256)
        	at org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan.setUpBeforeClass(TestTableInputFormatScan.java:83)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
        	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
        	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
        	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
        	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
        	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
        	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
        	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
        Caused by: org.apache.hadoop.yarn.YarnException: java.io.IOException: ResourceManager failed to start. Final state is INITED
        	at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:152)
        	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
        	... 34 more
        Caused by: java.io.IOException: ResourceManager failed to start. Final state is INITED
        	at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:146)
        	... 35 more
        
        Show
        Ashutosh Jindal added a comment - This is the exception that i got while running testcases. Some other testcases are also failing with the same error. ------------------------------------------------------------------------------- Test set: org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan ------------------------------------------------------------------------------- Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 177.132 sec <<< FAILURE! org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan Time elapsed: 0 sec <<< ERROR! org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.mapred.MiniMRCluster at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:67) at org.apache.hadoop.mapred.MiniMRCluster. <init> (MiniMRCluster.java:180) at org.apache.hadoop.mapred.MiniMRCluster. <init> (MiniMRCluster.java:170) at org.apache.hadoop.mapred.MiniMRCluster. <init> (MiniMRCluster.java:162) at org.apache.hadoop.mapred.MiniMRCluster. <init> (MiniMRCluster.java:154) at org.apache.hadoop.mapred.MiniMRCluster. <init> (MiniMRCluster.java:147) at org.apache.hadoop.mapred.MiniMRCluster. <init> (MiniMRCluster.java:140) at org.apache.hadoop.mapred.MiniMRCluster. <init> (MiniMRCluster.java:133) at org.apache.hadoop.mapred.MiniMRCluster. <init> (MiniMRCluster.java:128) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1269) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1256) at org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan.setUpBeforeClass(TestTableInputFormatScan.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) Caused by: org.apache.hadoop.yarn.YarnException: java.io.IOException: ResourceManager failed to start. Final state is INITED at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:152) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) ... 34 more Caused by: java.io.IOException: ResourceManager failed to start. Final state is INITED at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:146) ... 35 more
        Hide
        Ashutosh Jindal added a comment -

        I am also facing the same issue. Testcases using Map-Red minicluster are failing with the same kind of exception when run with 0.23 version

        Show
        Ashutosh Jindal added a comment - I am also facing the same issue. Testcases using Map-Red minicluster are failing with the same kind of exception when run with 0.23 version
        Hide
        Andrew Purtell added a comment -

        It's a maven activation that chooses the 0.23+ dependency set. You'll see it in the root POM.

        Show
        Andrew Purtell added a comment - It's a maven activation that chooses the 0.23+ dependency set. You'll see it in the root POM.
        Hide
        Arun C Murthy added a comment -

        Btw, for my education, -Dhadoop.profile just flips some code in hbase and doesn't mean we are getting different versions of hadoop-mapreduce, correct?

        Show
        Arun C Murthy added a comment - Btw, for my education, -Dhadoop.profile just flips some code in hbase and doesn't mean we are getting different versions of hadoop-mapreduce, correct?
        Hide
        Andrew Purtell added a comment -

        Anything that uses the minicluster fails. I only checked the output of a few but all had the same type and order of exceptions.

        Show
        Andrew Purtell added a comment - Anything that uses the minicluster fails. I only checked the output of a few but all had the same type and order of exceptions.
        Hide
        Arun C Murthy added a comment -

        Andy, I assume you are seeing more than one test fail with similar errors?

        Show
        Arun C Murthy added a comment - Andy, I assume you are seeing more than one test fail with similar errors?
        Hide
        Andrew Purtell added a comment -

        -Dhadoop.profile=24 gives the same result, the profiles seem to be identical .

        Show
        Andrew Purtell added a comment - -Dhadoop.profile=24 gives the same result, the profiles seem to be identical .

          People

          • Assignee:
            Jimmy Xiang
            Reporter:
            Andrew Purtell
          • Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development