Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.3
    • Component/s: None
    • Labels:
      None
    1. SQOOP-725.patch
      11 kB
      Hari Shreedharan
    2. SQOOP-725-2.patch
      11 kB
      Hari Shreedharan

      Issue Links

        Activity

        Hide
        hshreedharan Hari Shreedharan added a comment -

        This patch fixes the NoSuchMethodError.
        A similar patch fixed the issue I found in Flume.

        I'm running on a mac and the tests seem to not terminate - this seems to be some issue in the minicluster code, and not a compatibility issue. It'd be great if someone could apply this patch and see if it works on Linux.

        Show
        hshreedharan Hari Shreedharan added a comment - This patch fixes the NoSuchMethodError. A similar patch fixed the issue I found in Flume. I'm running on a mac and the tests seem to not terminate - this seems to be some issue in the minicluster code, and not a compatibility issue. It'd be great if someone could apply this patch and see if it works on Linux.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Forgot to mention this. To verify:

        • rm -rf ~/.ivy2 (This is required, else any previous version of hbase in the ivy cache will stay and will not pull the new version).
        • Clone hbase repo, checkout 0.94.2 tag, and build using mvn clean install -DskipTests -Dhadoop.profile=2.0
        • Apply patch to sqoop trunk and build using hadoopversion=200
        Show
        hshreedharan Hari Shreedharan added a comment - Forgot to mention this. To verify: rm -rf ~/.ivy2 (This is required , else any previous version of hbase in the ivy cache will stay and will not pull the new version). Clone hbase repo, checkout 0.94.2 tag, and build using mvn clean install -DskipTests -Dhadoop.profile=2.0 Apply patch to sqoop trunk and build using hadoopversion=200
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Seems like it is hitting gc during the run, at least it runs – it basically ends up pausing in the middle of the test

        Show
        hshreedharan Hari Shreedharan added a comment - Seems like it is hitting gc during the run, at least it runs – it basically ends up pausing in the middle of the test
        Hide
        hshreedharan Hari Shreedharan added a comment -

        So this patch fixes the tests - they are being run, but this is what the JStack looks like:

        "main" prio=5 tid=7fa142800000 nid=0x110391000 waiting on condition [11038f000]
           java.lang.Thread.State: TIMED_WAITING (sleeping)
        	at java.lang.Thread.sleep(Native Method)
        	at java.lang.Thread.sleep(Thread.java:302)
        	at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:328)
        	at org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:54)
        	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:179)
        	at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
        	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
        	- locked <77d5563f8> (a org.apache.hadoop.hbase.MasterAddressTracker)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:591)
        	- locked <77d528ec8> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:650)
        	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:110)
        	at org.apache.sqoop.mapreduce.HBaseImportJob.jobSetup(HBaseImportJob.java:143)
        	at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:209)
        	at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
        	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        	at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        	at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:45)
        	at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:215)
        	at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:230)
        	at com.cloudera.sqoop.hbase.TestHBaseImport.testBasicUsage(TestHBaseImport.java:38)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at junit.framework.TestCase.runTest(TestCase.java:168)
        	at junit.framework.TestCase.runBare(TestCase.java:134)
        	at junit.framework.TestResult$1.protect(TestResult.java:110)
        	at junit.framework.TestResult.runProtected(TestResult.java:128)
        	at junit.framework.TestResult.run(TestResult.java:113)
        	at junit.framework.TestCase.run(TestCase.java:124)
        	at junit.framework.TestSuite.runTest(TestSuite.java:232)
        	at junit.framework.TestSuite.run(TestSuite.java:227)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
        

        It looks like the test is not really able to run properly on the minicluster and may never have been able to. I am not familiar with the code in question here, but it looks like it ends up indefinitely for the master. Maybe this needs to be looked at: org.apache.sqoop.mapreduce.HBaseImportJob.jobSetup(HBaseImportJob.java:143).

        Show
        hshreedharan Hari Shreedharan added a comment - So this patch fixes the tests - they are being run, but this is what the JStack looks like: "main" prio=5 tid=7fa142800000 nid=0x110391000 waiting on condition [11038f000] java.lang. Thread .State: TIMED_WAITING (sleeping) at java.lang. Thread .sleep(Native Method) at java.lang. Thread .sleep( Thread .java:302) at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:328) at org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:54) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:179) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82) - locked <77d5563f8> (a org.apache.hadoop.hbase.MasterAddressTracker) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:591) - locked <77d528ec8> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:650) at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:110) at org.apache.sqoop.mapreduce.HBaseImportJob.jobSetup(HBaseImportJob.java:143) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:209) at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476) at org.apache.sqoop.Sqoop.run(Sqoop.java:145) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:45) at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:215) at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:230) at com.cloudera.sqoop.hbase.TestHBaseImport.testBasicUsage(TestHBaseImport.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911) It looks like the test is not really able to run properly on the minicluster and may never have been able to. I am not familiar with the code in question here, but it looks like it ends up indefinitely for the master. Maybe this needs to be looked at: org.apache.sqoop.mapreduce.HBaseImportJob.jobSetup(HBaseImportJob.java:143).
        Hide
        jarcec Jarek Jarcec Cecho added a comment - - edited

        Precise steps that I've done:

        svn checkout http://svn.apache.org/repos/asf/hbase/tags/0.94.2 hbase-0.94.2
        cd hbase-0.94.2
        rm -rf ~/.m2 ~/.ivy2
        mvn clean install -DskipTests -Dhadoop.profile=2.0
        
        cd /path/to/sqoop
        ant clean test -Dhadoopversion=200 -Dtestcase=TestHBaseImport                                                  
        

        Unfortunately as was suggested in previous comments this tests has timed out for me (after 20 minutes).

        Show
        jarcec Jarek Jarcec Cecho added a comment - - edited Precise steps that I've done: svn checkout http: //svn.apache.org/repos/asf/hbase/tags/0.94.2 hbase-0.94.2 cd hbase-0.94.2 rm -rf ~/.m2 ~/.ivy2 mvn clean install -DskipTests -Dhadoop.profile=2.0 cd /path/to/sqoop ant clean test -Dhadoopversion=200 -Dtestcase=TestHBaseImport Unfortunately as was suggested in previous comments this tests has timed out for me (after 20 minutes).
        Hide
        hshreedharan Hari Shreedharan added a comment -

        It looks like I'm hitting the same issue in hadoop-20 profile too.

        Show
        hshreedharan Hari Shreedharan added a comment - It looks like I'm hitting the same issue in hadoop-20 profile too.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        I'm not sure how to selectively disable certain tests when hadoop-23 and hadoop-2. If you do, please feel free to add to the patch and commit.

        Show
        hshreedharan Hari Shreedharan added a comment - I'm not sure how to selectively disable certain tests when hadoop-23 and hadoop-2. If you do, please feel free to add to the patch and commit.
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Hi Hari,
        it seems really working on profile 200. Thank you very much for your effort to make this happen, I'm excited to see HBase tests working again.

        I'm unfortunately getting compilation errors in profile 100:

            [javac] /home/jarcec/apache/repos/sqoop/src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java:230: cannot find symbol
            [javac] symbol  : method getTempDirectory()
            [javac] location: class org.apache.commons.io.FileUtils
            [javac]     File baseDir = FileUtils.getTempDirectory();
            [javac]                             ^
        

        Would you mind taking a look?

        Show
        jarcec Jarek Jarcec Cecho added a comment - Hi Hari, it seems really working on profile 200. Thank you very much for your effort to make this happen, I'm excited to see HBase tests working again. I'm unfortunately getting compilation errors in profile 100: [javac] /home/jarcec/apache/repos/sqoop/src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java:230: cannot find symbol [javac] symbol : method getTempDirectory() [javac] location: class org.apache.commons.io.FileUtils [javac] File baseDir = FileUtils.getTempDirectory(); [javac] ^ Would you mind taking a look?
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Looks like there is something wrong with the commons io versions - probably due to the different versions required by Hadoop. I will remove that dependency by using the java.io.tempdir system property

        Show
        hshreedharan Hari Shreedharan added a comment - Looks like there is something wrong with the commons io versions - probably due to the different versions required by Hadoop. I will remove that dependency by using the java.io.tempdir system property
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=33a7a81412d789371a235323ac15f875e1870894

        Thank you very much for your contribution Hari!

        Jarcec

        Show
        jarcec Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=33a7a81412d789371a235323ac15f875e1870894 Thank you very much for your contribution Hari! Jarcec
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        I've created SQOOP-731 to move out HBase tests to manual test set.

        Show
        jarcec Jarek Jarcec Cecho added a comment - I've created SQOOP-731 to move out HBase tests to manual test set.
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop20 #337 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/337/)
        SQOOP-725: Enable HBase tests in sqoop 1 (Revision 33a7a81412d789371a235323ac15f875e1870894)

        Result = SUCCESS
        jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=33a7a81412d789371a235323ac15f875e1870894
        Files :

        • src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java
        • build.xml
        • src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java
        • src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java
        • ivy/ivysettings.xml
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #337 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/337/ ) SQOOP-725 : Enable HBase tests in sqoop 1 (Revision 33a7a81412d789371a235323ac15f875e1870894) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=33a7a81412d789371a235323ac15f875e1870894 Files : src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java build.xml src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java ivy/ivysettings.xml
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop23 #506 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/506/)
        SQOOP-725: Enable HBase tests in sqoop 1 (Revision 33a7a81412d789371a235323ac15f875e1870894)

        Result = SUCCESS
        jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=33a7a81412d789371a235323ac15f875e1870894
        Files :

        • src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java
        • build.xml
        • src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java
        • src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java
        • ivy/ivysettings.xml
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #506 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/506/ ) SQOOP-725 : Enable HBase tests in sqoop 1 (Revision 33a7a81412d789371a235323ac15f875e1870894) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=33a7a81412d789371a235323ac15f875e1870894 Files : src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java build.xml src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java ivy/ivysettings.xml
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop200 #340 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/340/)
        SQOOP-725: Enable HBase tests in sqoop 1 (Revision 33a7a81412d789371a235323ac15f875e1870894)

        Result = FAILURE
        jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=33a7a81412d789371a235323ac15f875e1870894
        Files :

        • build.xml
        • ivy/ivysettings.xml
        • src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java
        • src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java
        • src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #340 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/340/ ) SQOOP-725 : Enable HBase tests in sqoop 1 (Revision 33a7a81412d789371a235323ac15f875e1870894) Result = FAILURE jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=33a7a81412d789371a235323ac15f875e1870894 Files : build.xml ivy/ivysettings.xml src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        The failure on profile hadoop 200 is expected and will be fixed by SQOOP-731. I'm surprised though to see the profile hadoop 23 succeeded as I was expecting it's failure as well.

        Show
        jarcec Jarek Jarcec Cecho added a comment - The failure on profile hadoop 200 is expected and will be fixed by SQOOP-731 . I'm surprised though to see the profile hadoop 23 succeeded as I was expecting it's failure as well.
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop100 #329 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/329/)
        SQOOP-725: Enable HBase tests in sqoop 1 (Revision 33a7a81412d789371a235323ac15f875e1870894)

        Result = FAILURE
        jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=33a7a81412d789371a235323ac15f875e1870894
        Files :

        • src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java
        • src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java
        • src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java
        • ivy/ivysettings.xml
        • build.xml
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #329 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/329/ ) SQOOP-725 : Enable HBase tests in sqoop 1 (Revision 33a7a81412d789371a235323ac15f875e1870894) Result = FAILURE jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=33a7a81412d789371a235323ac15f875e1870894 Files : src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java ivy/ivysettings.xml build.xml
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Jarek Jarcec Cecho Could you please re-run the hadoop-1 build? The build is weird, since some HBase tests are successful, and just one fails.

        Show
        hshreedharan Hari Shreedharan added a comment - Jarek Jarcec Cecho Could you please re-run the hadoop-1 build? The build is weird, since some HBase tests are successful, and just one fails.
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Yes of course, I've run it and it seems to be successful: https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/331/console

        Show
        jarcec Jarek Jarcec Cecho added a comment - Yes of course, I've run it and it seems to be successful: https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/331/console
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Thanks!

        Show
        hshreedharan Hari Shreedharan added a comment - Thanks!

          People

          • Assignee:
            hshreedharan Hari Shreedharan
            Reporter:
            hshreedharan Hari Shreedharan
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development