HBase
  1. HBase
  2. HBASE-6891 Hadoop 2 unit test failures
  3. HBASE-8453

TestImportExport failing again due to configuration issues

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.98.0, 0.94.8, 0.95.1
    • Fix Version/s: 0.98.0, 0.95.2, 0.94.9
    • Component/s: mapreduce, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestImportExport fails for me with a connection refused exception:

      java.lang.reflect.UndeclaredThrowableException
      	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
      	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:162)
      	at org.apache.hadoop.yarn.client.YarnClientImpl.getNewApplication(YarnClientImpl.java:121)
      	at org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:107)
      	at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:231)
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:352)
      [...]
      Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: Call From ip-10-174-75-236/10.174.75.236 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)
      	at com.sun.proxy.$Proxy89.getNewApplication(Unknown Source)
      	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:159)
      	... 42 more
      

      Settings in the MiniMRCluster configuration are not properly propagated in this test.

      1. 8453.patch
        3 kB
        Nick Dimiduk
      2. 8453.patch
        3 kB
        Andrew Purtell
      3. 8453-v1-0.94.patch
        8 kB
        Andrew Purtell

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          Attached patch fixes TestImportExport failures for me locally, tested against 2.0.4 with 0.94.7. It exposes the MiniMRCluster in HBaseTestingUtility, then in TestImportExport uses MiniMRCluster#createJobConf to create a JobConf for Import.createSubmittableJob or Export.createSubmittableJob, and HBaseConfiguration.merge to merge in our DFS and HDFS configurations into that JobConf.

          Show
          Andrew Purtell added a comment - Attached patch fixes TestImportExport failures for me locally, tested against 2.0.4 with 0.94.7. It exposes the MiniMRCluster in HBaseTestingUtility, then in TestImportExport uses MiniMRCluster#createJobConf to create a JobConf for Import.createSubmittableJob or Export.createSubmittableJob, and HBaseConfiguration.merge to merge in our DFS and HDFS configurations into that JobConf.
          Hide
          Lars Hofhansl added a comment -

          Looks good to me.

          Show
          Lars Hofhansl added a comment - Looks good to me.
          Hide
          Ted Yu added a comment -

          Using HBaseConfiguration.merge to merge in configuration would be the strategy for other tests which don't currently pass based on 2.0.4
          Should the HBaseConfiguration.merge call be placed in a common code path so that other tests can benefit ?

          Show
          Ted Yu added a comment - Using HBaseConfiguration.merge to merge in configuration would be the strategy for other tests which don't currently pass based on 2.0.4 Should the HBaseConfiguration.merge call be placed in a common code path so that other tests can benefit ?
          Hide
          Andrew Purtell added a comment -

          Thanks Lars. Will put this in through trunk first, tomorrow.

          Show
          Andrew Purtell added a comment - Thanks Lars. Will put this in through trunk first, tomorrow.
          Hide
          Andrew Purtell added a comment -

          Trunk patch, tested locally with 2.0.4-alpha, let's see if HadoopQA picks it up.

          Show
          Andrew Purtell added a comment - Trunk patch, tested locally with 2.0.4-alpha, let's see if HadoopQA picks it up.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581018/8453.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.util.TestHBaseFsck

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581018/8453.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.util.TestHBaseFsck Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5487//console This message is automatically generated.
          Hide
          Andrew Purtell added a comment -

          Cool, TestHBaseFsck may need a look but this patch is good. Will commit soon.

          Show
          Andrew Purtell added a comment - Cool, TestHBaseFsck may need a look but this patch is good. Will commit soon.
          Hide
          Andrew Purtell added a comment -

          Committed.

          Show
          Andrew Purtell added a comment - Committed.
          Hide
          Andrew Purtell added a comment -

          Reopening. This passed locally testing with Hadoop 1 and Hadoop 2, on a private Jenkins with 2.0.4, and on HadoopQA, but appears to have introduced regressions on EC2 Jenkins 0.94 build.

          Show
          Andrew Purtell added a comment - Reopening. This passed locally testing with Hadoop 1 and Hadoop 2, on a private Jenkins with 2.0.4, and on HadoopQA, but appears to have introduced regressions on EC2 Jenkins 0.94 build.
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #168 (See https://builds.apache.org/job/hbase-0.95/168/)
          HBASE-8453. TestImportExport failing again due to configuration issues (Revision 1477374)

          Result = SUCCESS
          apurtell :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #168 (See https://builds.apache.org/job/hbase-0.95/168/ ) HBASE-8453 . TestImportExport failing again due to configuration issues (Revision 1477374) Result = SUCCESS apurtell : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Andrew Purtell added a comment -

          It's not clear from the test output what the problem is, so I am going to revert.

          Show
          Andrew Purtell added a comment - It's not clear from the test output what the problem is, so I am going to revert.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #4086 (See https://builds.apache.org/job/HBase-TRUNK/4086/)
          HBASE-8453. TestImportExport failing again due to configuration issues (Revision 1477373)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #4086 (See https://builds.apache.org/job/HBase-TRUNK/4086/ ) HBASE-8453 . TestImportExport failing again due to configuration issues (Revision 1477373) Result = FAILURE apurtell : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          ramkrishna.s.vasudevan added a comment -

          The TestHBaseFsck failures are due to this patch? Are you checking this Andy?

          Show
          ramkrishna.s.vasudevan added a comment - The TestHBaseFsck failures are due to this patch? Are you checking this Andy?
          Hide
          Andrew Purtell added a comment -

          How would TestHaseFsck failures be related to this patch? I don't see it. Plus, it has been reverted already for another reason. See above comments.

          Show
          Andrew Purtell added a comment - How would TestHaseFsck failures be related to this patch? I don't see it. Plus, it has been reverted already for another reason. See above comments.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Yes Andy. I was surprised to see that TestHBaseFsck fails due to this.

          Show
          ramkrishna.s.vasudevan added a comment - Yes Andy. I was surprised to see that TestHBaseFsck fails due to this.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #970 (See https://builds.apache.org/job/HBase-0.94/970/)
          Revert HBASE-8453, HBASE-8454. (Revision 1477450)
          HBASE-8453. TestImportExport failing again due to configuration issues (Revision 1477375)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java

          apurtell :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #970 (See https://builds.apache.org/job/HBase-0.94/970/ ) Revert HBASE-8453 , HBASE-8454 . (Revision 1477450) HBASE-8453 . TestImportExport failing again due to configuration issues (Revision 1477375) Result = FAILURE apurtell : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java apurtell : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #85 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/85/)
          Revert HBASE-8453, HBASE-8454. (Revision 1477449)
          HBASE-8453. TestImportExport failing again due to configuration issues (Revision 1477374)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java

          apurtell :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #85 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/85/ ) Revert HBASE-8453 , HBASE-8454 . (Revision 1477449) HBASE-8453 . TestImportExport failing again due to configuration issues (Revision 1477374) Result = FAILURE apurtell : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java apurtell : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #515 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/515/)
          Revert HBASE-8453, HBASE-8454. (Revision 1477448)
          HBASE-8453. TestImportExport failing again due to configuration issues (Revision 1477373)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java

          apurtell :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #515 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/515/ ) Revert HBASE-8453 , HBASE-8454 . (Revision 1477448) HBASE-8453 . TestImportExport failing again due to configuration issues (Revision 1477373) Result = FAILURE apurtell : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java apurtell : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #169 (See https://builds.apache.org/job/hbase-0.95/169/)
          Revert HBASE-8453, HBASE-8454. (Revision 1477449)

          Result = SUCCESS
          apurtell :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #169 (See https://builds.apache.org/job/hbase-0.95/169/ ) Revert HBASE-8453 , HBASE-8454 . (Revision 1477449) Result = SUCCESS apurtell : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #4087 (See https://builds.apache.org/job/HBase-TRUNK/4087/)
          Revert HBASE-8453, HBASE-8454. (Revision 1477448)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #4087 (See https://builds.apache.org/job/HBase-TRUNK/4087/ ) Revert HBASE-8453 , HBASE-8454 . (Revision 1477448) Result = FAILURE apurtell : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java
          Hide
          Andrew Purtell added a comment -

          Unassigning for now, will circle back to this next week.

          Show
          Andrew Purtell added a comment - Unassigning for now, will circle back to this next week.
          Hide
          Jonathan Hsieh added a comment -

          I noticed an internal ip/hostname – could this be a ec2 internal-external ip reverse-dns problem?

          Show
          Jonathan Hsieh added a comment - I noticed an internal ip/hostname – could this be a ec2 internal-external ip reverse-dns problem?
          Hide
          Andrew Purtell added a comment -

          I noticed an internal ip/hostname – could this be a ec2 internal-external ip reverse-dns problem?

          That was my thought too since the failures only happened in an EC2 environment, but it hosed a Hadoop 1 build that was otherwise passing, so I reverted the change. Need to dump out configs before and after and eyeball differences.

          Show
          Andrew Purtell added a comment - I noticed an internal ip/hostname – could this be a ec2 internal-external ip reverse-dns problem? That was my thought too since the failures only happened in an EC2 environment, but it hosed a Hadoop 1 build that was otherwise passing, so I reverted the change. Need to dump out configs before and after and eyeball differences.
          Hide
          Andrew Purtell added a comment -

          I have found that getting a JobConf from the MiniMRCluster and then using HBaseConfiguration.merge to merge in HBase minicluster details will fix all remaining Hadoop 2 test failures involving mapreduce. It's just a matter of finding all of the places Job is instantaned and doing so. Unfortunately the result then fails for Hadoop 1 but only in EC2 environments. Might get back to that this week.

          Show
          Andrew Purtell added a comment - I have found that getting a JobConf from the MiniMRCluster and then using HBaseConfiguration.merge to merge in HBase minicluster details will fix all remaining Hadoop 2 test failures involving mapreduce. It's just a matter of finding all of the places Job is instantaned and doing so. Unfortunately the result then fails for Hadoop 1 but only in EC2 environments. Might get back to that this week.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #141 (See https://builds.apache.org/job/HBase-0.94-security/141/)
          Revert HBASE-8453, HBASE-8454. (Revision 1477450)
          HBASE-8453. TestImportExport failing again due to configuration issues (Revision 1477375)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java

          apurtell :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #141 (See https://builds.apache.org/job/HBase-0.94-security/141/ ) Revert HBASE-8453 , HBASE-8454 . (Revision 1477450) HBASE-8453 . TestImportExport failing again due to configuration issues (Revision 1477375) Result = FAILURE apurtell : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java apurtell : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Lars Hofhansl added a comment -

          Moving to 0.94.9.

          Show
          Lars Hofhansl added a comment - Moving to 0.94.9.
          Hide
          Andrew Purtell added a comment -

          I have found that exposing the MiniMRCluster out of HBaseTestingUtility, getting a JobConf from the MiniMRCluster instance and then using HBaseConfiguration.merge to merge in HBase minicluster details will fix all remaining Hadoop 2 test failures involving mapreduce, and also fixes Hadoop 2 related issues with HBASE-8746. The downside of this change is Hadoop 1 tests will not pass in EC2 environments. It is a test only issue. Worth using this strategy? (I'd argue on moving generally toward deprecating Hadoop 1.)

          Show
          Andrew Purtell added a comment - I have found that exposing the MiniMRCluster out of HBaseTestingUtility, getting a JobConf from the MiniMRCluster instance and then using HBaseConfiguration.merge to merge in HBase minicluster details will fix all remaining Hadoop 2 test failures involving mapreduce, and also fixes Hadoop 2 related issues with HBASE-8746 . The downside of this change is Hadoop 1 tests will not pass in EC2 environments. It is a test only issue. Worth using this strategy? (I'd argue on moving generally toward deprecating Hadoop 1.)
          Hide
          stack added a comment -

          I am with John that it would be better understanding why the failure but in interests of expediency I would be up for a patch that does the merge (John is otherwise occupied over next few weeks).

          The downside of this change is Hadoop 1 tests will not pass in EC2 environments.

          Why is this Andrew? How does it fail?

          Thanks.

          Show
          stack added a comment - I am with John that it would be better understanding why the failure but in interests of expediency I would be up for a patch that does the merge (John is otherwise occupied over next few weeks). The downside of this change is Hadoop 1 tests will not pass in EC2 environments. Why is this Andrew? How does it fail? Thanks.
          Hide
          stack added a comment -

          Andrew Purtell Does the patch here work?

          Show
          stack added a comment - Andrew Purtell Does the patch here work?
          Hide
          Andrew Purtell added a comment - - edited

          MR tests complete for both Hadoop 1 and 2 - except for on EC2. I do not know definitively why Hadoop 1's MiniMRCluster becomes unhappy on EC2 with this change. I could guess (similar to Jon's rumination) but that wouldn't add anything. This change appears to solve MR related test issues everywhere for Hadoop 2 without special casing YARN internal configs bound to change at every release. The downside is a test only quirk (this is the MRMiniCluster's problem - not a real MR issue on a real deploy) with Hadoop 1 on EC2.

          I'm fine with holding off on committing this until someone can find the time to dump configs at every step and eyeball them. Not sure when I'll have that time personally. Thought I'd ping here again since I found another place where this fixes Hadoop 2 issues today on HBASE-8746. (And I no longer care about Hadoop 1.)

          Show
          Andrew Purtell added a comment - - edited MR tests complete for both Hadoop 1 and 2 - except for on EC2. I do not know definitively why Hadoop 1's MiniMRCluster becomes unhappy on EC2 with this change. I could guess (similar to Jon's rumination) but that wouldn't add anything. This change appears to solve MR related test issues everywhere for Hadoop 2 without special casing YARN internal configs bound to change at every release. The downside is a test only quirk (this is the MRMiniCluster's problem - not a real MR issue on a real deploy) with Hadoop 1 on EC2. I'm fine with holding off on committing this until someone can find the time to dump configs at every step and eyeball them. Not sure when I'll have that time personally. Thought I'd ping here again since I found another place where this fixes Hadoop 2 issues today on HBASE-8746 . (And I no longer care about Hadoop 1.)
          Hide
          stack added a comment -

          How about committing this for now and opening a new issue to cover the ec2 failures? At a minimum we are narrowing the failure some.

          Show
          stack added a comment - How about committing this for now and opening a new issue to cover the ec2 failures? At a minimum we are narrowing the failure some.
          Hide
          Nick Dimiduk added a comment -

          I'm +1 for committing this and opening a new ticket for ec2 failures.

          I'm also curious under what conditions we're willing to bump the hadoop-2 dependency to 2.0.5-alpha. I believe there are a number of HDFS and MR bugs fixed in that release and I expect we'll have better stability on that version.

          I'll see if I can reproduce and do the configuration diff described by Andrew Purtell.

          Show
          Nick Dimiduk added a comment - I'm +1 for committing this and opening a new ticket for ec2 failures. I'm also curious under what conditions we're willing to bump the hadoop-2 dependency to 2.0.5-alpha. I believe there are a number of HDFS and MR bugs fixed in that release and I expect we'll have better stability on that version. I'll see if I can reproduce and do the configuration diff described by Andrew Purtell .
          Hide
          Nick Dimiduk added a comment -

          I've applied this patch against trunk and run in an EC2 vm (Linux ip-10-152-187-175 2.6.32-276.el6.x86_64 #1 SMP Tue May 29 17:38:19 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux, ami-89ad18e0, jdk-7u21-linux-x64). The test passes on both hadoop1 and hadoop2 profiles.

          Show
          Nick Dimiduk added a comment - I've applied this patch against trunk and run in an EC2 vm (Linux ip-10-152-187-175 2.6.32-276.el6.x86_64 #1 SMP Tue May 29 17:38:19 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux, ami-89ad18e0, jdk-7u21-linux-x64). The test passes on both hadoop1 and hadoop2 profiles.
          Hide
          Nick Dimiduk added a comment -

          rebased against trunk.

          Show
          Nick Dimiduk added a comment - rebased against trunk.
          Hide
          Nick Dimiduk added a comment -

          The 0.94 patch applies cleanly and runs just fine for both hadoop1 and hadoop2 profiles on the same VM.

          Show
          Nick Dimiduk added a comment - The 0.94 patch applies cleanly and runs just fine for both hadoop1 and hadoop2 profiles on the same VM.
          Hide
          stack added a comment -

          Committed to 0.94, 0.95 and trunk. Thanks Nick for driving this one home. I'll wait on opening an EC2 issue till we have an actual failure (given Nicks testing done above).

          Show
          stack added a comment - Committed to 0.94, 0.95 and trunk. Thanks Nick for driving this one home. I'll wait on opening an EC2 issue till we have an actual failure (given Nicks testing done above).
          Hide
          Andrew Purtell added a comment -

          Great!

          Show
          Andrew Purtell added a comment - Great!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12588424/8453.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6061//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588424/8453.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6061//console This message is automatically generated.
          Hide
          Nick Dimiduk added a comment -

          oops, I failed to resolve message order before modifying ticket state.

          Show
          Nick Dimiduk added a comment - oops, I failed to resolve message order before modifying ticket state.
          Hide
          Nick Dimiduk added a comment -

          The commit went in under the parent ticket and produced this build error on 0.94-security.

          Show
          Nick Dimiduk added a comment - The commit went in under the parent ticket and produced this build error on 0.94-security.
          Hide
          stack added a comment -

          Nick Dimiduk I should back out the 0.94 patch? Let me fix the commit log.

          Show
          stack added a comment - Nick Dimiduk I should back out the 0.94 patch? Let me fix the commit log.
          Hide
          Andrew Purtell added a comment -

          This is the familiar pattern:

          I've applied this patch against trunk and run in an EC2 vm (Linux ip-10-152-187-175 2.6.32-276.el6.x86_64 #1 SMP Tue May 29 17:38:19 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux, ami-89ad18e0, jdk-7u21-linux-x64). The test passes on both hadoop1 and hadoop2 profiles.

          and

          The 0.94 patch applies cleanly and runs just fine for both hadoop1 and hadoop2 profiles on the same VM.

          But then we have this:

          produced this build error on 0.94-security.

          A worse result than before. Not confined to EC2.

          Let's revert the patch and close this issue definitively.

          I propose instead we move MiniMRCluster instantiation and JobConf building and its gory details to the hadoop-compat modules. A simple consistent approach remains elusive.

          Show
          Andrew Purtell added a comment - This is the familiar pattern: I've applied this patch against trunk and run in an EC2 vm (Linux ip-10-152-187-175 2.6.32-276.el6.x86_64 #1 SMP Tue May 29 17:38:19 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux, ami-89ad18e0, jdk-7u21-linux-x64). The test passes on both hadoop1 and hadoop2 profiles. and The 0.94 patch applies cleanly and runs just fine for both hadoop1 and hadoop2 profiles on the same VM. But then we have this: produced this build error on 0.94-security. A worse result than before. Not confined to EC2. Let's revert the patch and close this issue definitively. I propose instead we move MiniMRCluster instantiation and JobConf building and its gory details to the hadoop-compat modules. A simple consistent approach remains elusive.
          Hide
          Nick Dimiduk added a comment -

          I'm running locally vs 0.94+security profile. Will run a couple times to rule out flakey test.

          Show
          Nick Dimiduk added a comment - I'm running locally vs 0.94+security profile. Will run a couple times to rule out flakey test.
          Hide
          Nick Dimiduk added a comment -

          The test fails consistently for me with mvn -Psecurity -PlocalTests test -Dtest=org.apache.hadoop.hbase.mapreduce.TestImportExport. I call that progress

          Show
          Nick Dimiduk added a comment - The test fails consistently for me with mvn -Psecurity -PlocalTests test -Dtest=org.apache.hadoop.hbase.mapreduce.TestImportExport . I call that progress
          Hide
          stack added a comment -

          Nick Dimiduk You going to dig in? Else let me revert as per Andrew suggestion.

          Show
          stack added a comment - Nick Dimiduk You going to dig in? Else let me revert as per Andrew suggestion.
          Hide
          Nick Dimiduk added a comment -

          I'm not digging in today.

          Show
          Nick Dimiduk added a comment - I'm not digging in today.
          Hide
          stack added a comment -

          Reverted patch from 0.94, 0.95, and trunk. Resolving as won't fix. Lets fix hadoop2 failures doing as Andrew suggests by making adding in hadoop1 and hadoop2 compat modules.

          Show
          stack added a comment - Reverted patch from 0.94, 0.95, and trunk. Resolving as won't fix. Lets fix hadoop2 failures doing as Andrew suggests by making adding in hadoop1 and hadoop2 compat modules.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #170 (See https://builds.apache.org/job/HBase-0.94-security/170/)
          HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494349)

          Result = SUCCESS
          stack :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #170 (See https://builds.apache.org/job/HBase-0.94-security/170/ ) HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494349) Result = SUCCESS stack : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #136 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/136/)
          HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494350)
          HBASE-8453 TestImportExport failing again due to configuration issues (Revision 1494266)

          Result = FAILURE
          stack :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java

          stack :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #136 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/136/ ) HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494350) HBASE-8453 TestImportExport failing again due to configuration issues (Revision 1494266) Result = FAILURE stack : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java stack : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #571 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/571/)
          HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494351)
          HBASE-8453 TestImportExport failing again due to configuration issues (Revision 1494256)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java

          stack :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #571 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/571/ ) HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494351) HBASE-8453 TestImportExport failing again due to configuration issues (Revision 1494256) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java stack : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #1015 (See https://builds.apache.org/job/HBase-0.94/1015/)
          HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494349)

          Result = SUCCESS
          stack :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #1015 (See https://builds.apache.org/job/HBase-0.94/1015/ ) HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494349) Result = SUCCESS stack : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #247 (See https://builds.apache.org/job/hbase-0.95/247/)
          HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494350)

          Result = SUCCESS
          stack :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #247 (See https://builds.apache.org/job/hbase-0.95/247/ ) HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494350) Result = SUCCESS stack : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #4185 (See https://builds.apache.org/job/HBase-TRUNK/4185/)
          HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494351)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #4185 (See https://builds.apache.org/job/HBase-TRUNK/4185/ ) HBASE-8453 TestImportExport failing again due to configuration issues – REVERT (Revision 1494351) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java

            People

            • Assignee:
              Unassigned
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development