HBase
  1. HBase
  2. HBASE-5876

TestImportExport has been failing against hadoop 0.23 profile

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0, 0.95.2
    • Fix Version/s: 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestImportExport has been failing against hadoop 0.23 profile

      1. hbase-5876.patch
        4 kB
        Jonathan Hsieh
      2. hbase-5876-94.patch
        2 kB
        Jonathan Hsieh
      3. hbase-5876-94-v3.patch
        10 kB
        Jonathan Hsieh
      4. hbase-5876-trunk-v3.patch
        10 kB
        Jonathan Hsieh
      5. hbase-5876-v2.patch
        5 kB
        Jonathan Hsieh

        Issue Links

          Activity

          Hide
          Uma Maheswara Rao G added a comment -

          2012-04-26 01:23:47,790 ERROR [main] common.Util(58): Syntax error in URI E:\Repoitories\Hbase\target\test-data\79fa72c8-f019-4c29-be3d-cc67230f70cd\dfscluster_177a4b0f-5ebc-4dfc-9d89-867fefec2c6a\dfs\name2. Please check hdfs configuration.
          java.net.URISyntaxException: Illegal character in opaque part at index 2: E:\Repoitories\Hbase\target\test-data\79fa72c8-f019-4c29-be3d-cc67230f70cd\dfscluster_177a4b0f-5ebc-4dfc-9d89-867fefec2c6a\dfs\name2
          at java.net.URI$Parser.fail(Unknown Source)
          at java.net.URI$Parser.checkChars(Unknown Source)
          at java.net.URI$Parser.parse(Unknown Source)
          at java.net.URI.<init>(Unknown Source)
          at org.apache.hadoop.hdfs.server.common.Util.stringAsURI(Util.java:56)
          at org.apache.hadoop.hdfs.server.common.Util.stringCollectionAsURIs(Util.java:106)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStorageDirs(FSNamesystem.java:761)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceEditsDirs(FSNamesystem.java:806)

          URI creation is failing with passed directory here. Let me take a look.

          Show
          Uma Maheswara Rao G added a comment - 2012-04-26 01:23:47,790 ERROR [main] common.Util(58): Syntax error in URI E:\Repoitories\Hbase\target\test-data\79fa72c8-f019-4c29-be3d-cc67230f70cd\dfscluster_177a4b0f-5ebc-4dfc-9d89-867fefec2c6a\dfs\name2. Please check hdfs configuration. java.net.URISyntaxException: Illegal character in opaque part at index 2: E:\Repoitories\Hbase\target\test-data\79fa72c8-f019-4c29-be3d-cc67230f70cd\dfscluster_177a4b0f-5ebc-4dfc-9d89-867fefec2c6a\dfs\name2 at java.net.URI$Parser.fail(Unknown Source) at java.net.URI$Parser.checkChars(Unknown Source) at java.net.URI$Parser.parse(Unknown Source) at java.net.URI.<init>(Unknown Source) at org.apache.hadoop.hdfs.server.common.Util.stringAsURI(Util.java:56) at org.apache.hadoop.hdfs.server.common.Util.stringCollectionAsURIs(Util.java:106) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStorageDirs(FSNamesystem.java:761) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceEditsDirs(FSNamesystem.java:806) URI creation is failing with passed directory here. Let me take a look.
          Hide
          Uma Maheswara Rao G added a comment -

          @Ted, is it the same issue you are facing?
          This failure is on my windows box. Actually problem due to backward slash in windows i think. Simple URI creation also fails with the same reason. Once I changed it to forward slash '/' , it started working.

          Show
          Uma Maheswara Rao G added a comment - @Ted, is it the same issue you are facing? This failure is on my windows box. Actually problem due to backward slash in windows i think. Simple URI creation also fails with the same reason. Once I changed it to forward slash '/' , it started working.
          Show
          Ted Yu added a comment - I face different issue on Macbook. See: https://issues.apache.org/jira/browse/HBASE-5870?focusedCommentId=13261773&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13261773
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks a lot, Jon for signing up this. Actually I thought to take a look on this weekend. Unfortunately I may not be able to look on this weekend as came out of station. Thanks for taking this

          Show
          Uma Maheswara Rao G added a comment - Thanks a lot, Jon for signing up this. Actually I thought to take a look on this weekend. Unfortunately I may not be able to look on this weekend as came out of station. Thanks for taking this
          Hide
          Jonathan Hsieh added a comment -

          I started going down the path and chatting with alejandro – one of the guys involved with MAPREDUCE-3169 (new MiniMRCluster for 23). It is likely a problem with job submission and will take a little digging.

          Show
          Jonathan Hsieh added a comment - I started going down the path and chatting with alejandro – one of the guys involved with MAPREDUCE-3169 (new MiniMRCluster for 23). It is likely a problem with job submission and will take a little digging.
          Hide
          Ted Yu added a comment -

          From https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/156/testReport/org.apache.hadoop.hbase.mapreduce/TestImportTsv/testMROnTableWithCustomMapper/:

          2012-04-27 00:47:47,745 ERROR [pool-1-thread-1] service.CompositeService(72): Error starting services org.apache.hadoop.mapred.MiniMRCluster
          org.apache.hadoop.yarn.YarnException: java.io.IOException: ResourceManager failed to start. Final state is INITED
          	at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:152)
          	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
          	at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:67)
          	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180)
          	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170)
          	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162)
          	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:154)
          	at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:147)
          

          From https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/156/testReport/org.apache.hadoop.hbase.mapreduce/TestImportExport/testWithDeletes/:

          java.io.FileNotFoundException: File does not exist: /home/hudson/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/0.23.2-SNAPSHOT/hadoop-mapreduce-client-core-0.23.2-SNAPSHOT.jar
          	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:729)
          	at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
          	at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
          	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:246)
          	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:284)
          	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:355)
          	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
          	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
          	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
          	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1239)
          	at org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithDeletes(TestImportExport.java:181)
          
          Show
          Ted Yu added a comment - From https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/156/testReport/org.apache.hadoop.hbase.mapreduce/TestImportTsv/testMROnTableWithCustomMapper/: 2012-04-27 00:47:47,745 ERROR [pool-1-thread-1] service.CompositeService(72): Error starting services org.apache.hadoop.mapred.MiniMRCluster org.apache.hadoop.yarn.YarnException: java.io.IOException: ResourceManager failed to start. Final state is INITED at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:152) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:67) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:154) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:147) From https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/156/testReport/org.apache.hadoop.hbase.mapreduce/TestImportExport/testWithDeletes/: java.io.FileNotFoundException: File does not exist: /home/hudson/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/0.23.2-SNAPSHOT/hadoop-mapreduce-client-core-0.23.2-SNAPSHOT.jar at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:729) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:246) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:284) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:355) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1239) at org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithDeletes(TestImportExport.java:181)
          Hide
          Jonathan Hsieh added a comment -

          The first problem is that when running under the hadoop23 some how the TestImportExport job uses the "local" mapreduce.framework.name which assumes and eventually tries to read from a LocalFileSystem. It should use the "yarn" mapreduce.framework.name which will read from the instantiated minimrcluster/minidfs cluster.

          This gets past the job submission step but test still fails later on.

          Work continues.

          Show
          Jonathan Hsieh added a comment - The first problem is that when running under the hadoop23 some how the TestImportExport job uses the "local" mapreduce.framework.name which assumes and eventually tries to read from a LocalFileSystem. It should use the "yarn" mapreduce.framework.name which will read from the instantiated minimrcluster/minidfs cluster. This gets past the job submission step but test still fails later on. Work continues.
          Hide
          Jonathan Hsieh added a comment -

          This test was introduced for HBase 0.94 in HBASE-4682 and seems to have never succeeded against 0.23 hadoop.

          Show
          Jonathan Hsieh added a comment - This test was introduced for HBase 0.94 in HBASE-4682 and seems to have never succeeded against 0.23 hadoop.
          Hide
          Jonathan Hsieh added a comment -

          This is related to HBASE-5697 – it has to do with some sort of inconsistencies with old mr properties (fs.default.name) and new mr properties (fs.defaultFs). When only using the new hadoop 23 properties, this test passes consistently.

          Show
          Jonathan Hsieh added a comment - This is related to HBASE-5697 – it has to do with some sort of inconsistencies with old mr properties (fs.default.name) and new mr properties (fs.defaultFs). When only using the new hadoop 23 properties, this test passes consistently.
          Hide
          Jonathan Hsieh added a comment -

          Problems in previous code:

          1. yarn execution framework not used because HBaseTestingUtility.startMiniCluster().getConfiguration() used instead of the HBaseTestingUtility.getConfiguration().
          2. hadoop 1's mapred.output.dir and hadoop 2's fileoutputformat.outputdir caused export job's data to get lost.

          Currently running full builds against hadoop 1.0 and hadoop 0.23.x.

          Show
          Jonathan Hsieh added a comment - Problems in previous code: yarn execution framework not used because HBaseTestingUtility.startMiniCluster().getConfiguration() used instead of the HBaseTestingUtility.getConfiguration(). hadoop 1's mapred.output.dir and hadoop 2's fileoutputformat.outputdir caused export job's data to get lost. Currently running full builds against hadoop 1.0 and hadoop 0.23.x.
          Hide
          Jonathan Hsieh added a comment -

          This works against hadoop 1 and hadoop 23, but isn't pretty (manual selection of yarn).

          Other tests use the configuration from HBaseTestingUtility instead of the conf associated with the MiniCluster.

          I spent a bit of time trying to fold this in to use HBaseTestingUtlity but it seems to be have problems with the mapred.local.dir and mapred.job.tracker.

          Show
          Jonathan Hsieh added a comment - This works against hadoop 1 and hadoop 23, but isn't pretty (manual selection of yarn). Other tests use the configuration from HBaseTestingUtility instead of the conf associated with the MiniCluster. I spent a bit of time trying to fold this in to use HBaseTestingUtlity but it seems to be have problems with the mapred.local.dir and mapred.job.tracker.
          Hide
          Ted Yu added a comment -
          +      // o.a.h.mapreduce.lib.output FileOutputFormat.OUTDIR
          +      return "mapreduce.output.fileoutputformat.outputdir";
          

          Does the first line above note where the constant comes from ?

          Show
          Ted Yu added a comment - + // o.a.h.mapreduce.lib.output FileOutputFormat.OUTDIR + return "mapreduce.output.fileoutputformat.outputdir" ; Does the first line above note where the constant comes from ?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525533/hbase-5876.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1757//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1757//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1757//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525533/hbase-5876.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1757//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1757//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1757//console This message is automatically generated.
          Hide
          stack added a comment -

          +1 on patch

          Show
          stack added a comment - +1 on patch
          Hide
          Jonathan Hsieh added a comment -

          @Ted. Yes. If I actually use the constant the code will not compile under hadoop 1.x. I'll add to the comment about that before I commit.

          Show
          Jonathan Hsieh added a comment - @Ted. Yes. If I actually use the constant the code will not compile under hadoop 1.x. I'll add to the comment about that before I commit.
          Hide
          Jonathan Hsieh added a comment -

          0.94 branch (around rc3 time frame) compiles but has failing TestImportExport unit test against hadoop 23. This patch for 0.94 fixes that.

          Show
          Jonathan Hsieh added a comment - 0.94 branch (around rc3 time frame) compiles but has failing TestImportExport unit test against hadoop 23. This patch for 0.94 fixes that.
          Hide
          Jonathan Hsieh added a comment -

          v2 of the trunk patch includes better comments about where some of the magic property strings come from.

          Show
          Jonathan Hsieh added a comment - v2 of the trunk patch includes better comments about where some of the magic property strings come from.
          Hide
          Jonathan Hsieh added a comment -

          Uploaded correct 0.94 version of the patch. This patch didn't need to JobConf stuff – makes me thing that the trunk one could be simplified and that another patch forced it to be more complicated.

          Show
          Jonathan Hsieh added a comment - Uploaded correct 0.94 version of the patch. This patch didn't need to JobConf stuff – makes me thing that the trunk one could be simplified and that another patch forced it to be more complicated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525556/hbase-5876-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1760//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1760//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1760//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525556/hbase-5876-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1760//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1760//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1760//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525558/hbase-5876-94.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
          org.apache.hadoop.hbase.master.TestAssignmentManager

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1761//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1761//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1761//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525558/hbase-5876-94.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks org.apache.hadoop.hbase.master.TestAssignmentManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1761//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1761//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1761//console This message is automatically generated.
          Hide
          Jonathan Hsieh added a comment -

          Thanks for reviews ted and stack. Committed to 0.94 and 0.96/trunk.

          Show
          Jonathan Hsieh added a comment - Thanks for reviews ted and stack. Committed to 0.94 and 0.96/trunk.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #178 (See https://builds.apache.org/job/HBase-0.94/178/)
          HBASE-5876 TestImportExport has been failing against hadoop 0.23 profile (Revision 1333777)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #178 (See https://builds.apache.org/job/HBase-0.94/178/ ) HBASE-5876 TestImportExport has been failing against hadoop 0.23 profile (Revision 1333777) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2847 (See https://builds.apache.org/job/HBase-TRUNK/2847/)
          HBASE-5876 TestImportExport has been failing against hadoop 0.23 profile (Revision 1333778)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2847 (See https://builds.apache.org/job/HBase-TRUNK/2847/ ) HBASE-5876 TestImportExport has been failing against hadoop 0.23 profile (Revision 1333778) Result = FAILURE jmhsieh : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Ted Yu added a comment - Strange, I saw this after the fix went in: https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/167/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestImportExport/testSimpleCase/
          Hide
          Jonathan Hsieh added a comment -

          Ted noticed that the test is still failing. Reopening and reinvestigating.

          Show
          Jonathan Hsieh added a comment - Ted noticed that the test is still failing. Reopening and reinvestigating.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #26 (See https://builds.apache.org/job/HBase-0.94-security/26/)
          HBASE-5876 TestImportExport has been failing against hadoop 0.23 profile (Revision 1333777)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #26 (See https://builds.apache.org/job/HBase-0.94-security/26/ ) HBASE-5876 TestImportExport has been failing against hadoop 0.23 profile (Revision 1333777) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #192 (See https://builds.apache.org/job/HBase-TRUNK-security/192/)
          HBASE-5876 TestImportExport has been failing against hadoop 0.23 profile (Revision 1333778)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #192 (See https://builds.apache.org/job/HBase-TRUNK-security/192/ ) HBASE-5876 TestImportExport has been failing against hadoop 0.23 profile (Revision 1333778) Result = SUCCESS jmhsieh : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          Hide
          Jonathan Hsieh added a comment - - edited

          Interesting, the old version of TestImportExport against hadoop 1.0 (before my previous attempt) used the LocalMRRunner instead of the RPC to the MiniMRCluster. TestImportTsv always uses the RPC/MiniMRCluster.

          Use of the localmrrunner when miniMRcluster is spun up seems wrong. I'm going to force usage of the minimrcluster/rpc/runner in TestImportExport.

          Show
          Jonathan Hsieh added a comment - - edited Interesting, the old version of TestImportExport against hadoop 1.0 (before my previous attempt) used the LocalMRRunner instead of the RPC to the MiniMRCluster. TestImportTsv always uses the RPC/MiniMRCluster. Use of the localmrrunner when miniMRcluster is spun up seems wrong. I'm going to force usage of the minimrcluster/rpc/runner in TestImportExport.
          Hide
          Lars Hofhansl added a comment -

          Change for this issue were committed to both 0.96 and 0.94. Is this fixed, or should we close this issue?

          Show
          Lars Hofhansl added a comment - Change for this issue were committed to both 0.96 and 0.94. Is this fixed, or should we close this issue?
          Hide
          Jonathan Hsieh added a comment -

          I believe this is still broken – and I haven't gotten back to fixing it. I believe the export part of the test wrote to either hdfs/localfs and then the re-import part tried to read from the other, causing the test to fail. IIRC something happened with the mini mr dirs between 0.92 and 0.94 which broke the test.

          I'll give it another stab, and I don't get it done this weekend I'll revert it.

          Show
          Jonathan Hsieh added a comment - I believe this is still broken – and I haven't gotten back to fixing it. I believe the export part of the test wrote to either hdfs/localfs and then the re-import part tried to read from the other, causing the test to fail. IIRC something happened with the mini mr dirs between 0.92 and 0.94 which broke the test. I'll give it another stab, and I don't get it done this weekend I'll revert it.
          Hide
          Jonathan Hsieh added a comment -

          An update: I have a patch that works with both 0.94 (default 1.0.x and -Dhadoop.profile=23) and trunk (default 1.0.x and -Dhadoop.profile=2.0) but reopens the problem of having partition dirs in the project root directory instead of the target dir.

          I'm looking into fixing that problem now – I believe it is hadoop.tmp.dir and using oozie's minicluster setup as an examplar
          https://github.com/apache/oozie/blob/trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java#L656

          Show
          Jonathan Hsieh added a comment - An update: I have a patch that works with both 0.94 (default 1.0.x and -Dhadoop.profile=23) and trunk (default 1.0.x and -Dhadoop.profile=2.0) but reopens the problem of having partition dirs in the project root directory instead of the target dir. I'm looking into fixing that problem now – I believe it is hadoop.tmp.dir and using oozie's minicluster setup as an examplar https://github.com/apache/oozie/blob/trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java#L656
          Hide
          Jonathan Hsieh added a comment -

          Updated patches that have tests passing for 0.94 and trunk. (0.92 was good).

          Show
          Jonathan Hsieh added a comment - Updated patches that have tests passing for 0.94 and trunk. (0.92 was good).
          Hide
          ramkrishna.s.vasudevan added a comment -

          I tried running the patch for 0.94 on hadoop 2.0. It passed(but not much aware of the changes).

          Show
          ramkrishna.s.vasudevan added a comment - I tried running the patch for 0.94 on hadoop 2.0. It passed(but not much aware of the changes).
          Hide
          Jonathan Hsieh added a comment -

          Its been a few days, I'm going to commit later today unless I hear anything suggesting not to.

          Show
          Jonathan Hsieh added a comment - Its been a few days, I'm going to commit later today unless I hear anything suggesting not to.
          Hide
          Jonathan Hsieh added a comment -

          Since its been along time since other parts of this code was committed, I'm going to complete this as HBASE-6330, and close this one out.

          Show
          Jonathan Hsieh added a comment - Since its been along time since other parts of this code was committed, I'm going to complete this as HBASE-6330 , and close this one out.

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development