Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.2
    • Component/s: None
    • Labels:
      None

      Description

      This is a regression from "SQOOP-519: Update the version of hsqldb decadency to 2.x". Even though we set HADOOP_USER_CLASSPATH_FIRST in sqoop-configure.sh as discussed in SQOOP-524, tasks still pick up Hsqldb-1.8.x that is present in classpath.

      12/08/10 11:54:28 INFO mapred.JobClient: Task Id : attempt_201208101150_0001_m_000000_0, Status : FAILED
      Error: org.hsqldb.DatabaseURL.parseURL(Ljava/lang/String;ZZ)Lorg/hsqldb/persist/HsqlProperties;
      12/08/10 11:54:32 INFO mapred.JobClient: Task Id : attempt_201208101150_0001_m_000000_1, Status : FAILED
      Error: org.hsqldb.DatabaseURL.parseURL(Ljava/lang/String;ZZ)Lorg/hsqldb/persist/HsqlProperties;
      12/08/10 11:54:35 INFO mapred.JobClient: Task Id : attempt_201208101150_0001_m_000000_2, Status : FAILED
      Error: org.hsqldb.DatabaseURL.parseURL(Ljava/lang/String;ZZ)Lorg/hsqldb/persist/HsqlProperties;
      12/08/10 11:54:39 INFO mapred.JobClient: Job complete: job_201208101150_0001
      

      Hadoop 2.0 moved the hsqldb-2.x jar out of Hadoop classpath. (Please see MAPREDUCE-4498 for details.) So we no longer have the hsqldb-2.x jar in Hadoop classpath, and upgrading to hsqldb-2.x is no longer needed at all. So I suggest that we should revert SQOOP-519 and SQOOP-524 to avoid any potential issues due to conflicting hsqldb jars in classpath.

      1. SQOOP-571.patch
        17 kB
        Cheolsoo Park

        Activity

        Hide
        Cheolsoo Park added a comment -
        Show
        Cheolsoo Park added a comment - Review board: https://reviews.apache.org/r/6559/
        Hide
        Abhijeet Gaikwad added a comment -

        I have few queries:

        1. Are you running this on trunk?
        MAPREDUCE-4498 is in trunk and is not yet out, I feel we should not worry about it until the next Hadoop release is out.
        2. According to MAPREDUCE-4498, they have removed hsqldb from hadoop classpath, how is it that hsqldb-1.8.0 is getting picked up? - It is neither in hadoop nor in sqoop's classpath - let me know if I am missing anything here.

        Show
        Abhijeet Gaikwad added a comment - I have few queries: 1. Are you running this on trunk? MAPREDUCE-4498 is in trunk and is not yet out, I feel we should not worry about it until the next Hadoop release is out. 2. According to MAPREDUCE-4498 , they have removed hsqldb from hadoop classpath, how is it that hsqldb-1.8.0 is getting picked up? - It is neither in hadoop nor in sqoop's classpath - let me know if I am missing anything here.
        Hide
        Cheolsoo Park added a comment -

        Hi Abhijeet,

        To answer your questions,
        1) True, MAPRECUDE-4498 is not out yet, but it's going to be backported to the next release of CDH4 that is coming pretty soon.
        2) Hsqldb-1.8.0 is shipped by MR1. If you look at CDH4.0 for example, you will find it under /usr/lib/hadoop-0.20-mapreduce/lib.

        I must admit that the hadoop distro that I am having trouble with is CDH, but these are equally applicable to to any hadoop distro that ships hsqldb-1.8.x in it.

        Thanks!

        Show
        Cheolsoo Park added a comment - Hi Abhijeet, To answer your questions, 1) True, MAPRECUDE-4498 is not out yet, but it's going to be backported to the next release of CDH4 that is coming pretty soon. 2) Hsqldb-1.8.0 is shipped by MR1. If you look at CDH4.0 for example, you will find it under /usr/lib/hadoop-0.20-mapreduce/lib. I must admit that the hadoop distro that I am having trouble with is CDH, but these are equally applicable to to any hadoop distro that ships hsqldb-1.8.x in it. Thanks!
        Hide
        Cheolsoo Park added a comment -

        To be clear, I am running against CDH4.0 that is already shipping hsqldb-1.8.x. The original reason why I bumped the version of hsqdb to 2.x was because MAPREDUCE-4197 introduced hsqldb-2.x in MR2. But now they reverted their change in MAPREDUCE-4498, so there's no need to update it in sqoop.

        I should have thought about backward-incompatibilities in the first place, but I was under impression that Hadoop is moving to hsqldb-2.x all together. But I was terribly wrong.

        Does this make more sense?

        Show
        Cheolsoo Park added a comment - To be clear, I am running against CDH4.0 that is already shipping hsqldb-1.8.x. The original reason why I bumped the version of hsqdb to 2.x was because MAPREDUCE-4197 introduced hsqldb-2.x in MR2. But now they reverted their change in MAPREDUCE-4498 , so there's no need to update it in sqoop. I should have thought about backward-incompatibilities in the first place, but I was under impression that Hadoop is moving to hsqldb-2.x all together. But I was terribly wrong. Does this make more sense?
        Hide
        Jarek Jarcec Cecho added a comment -

        I've took different view on the issue. I've downloaded all 0.23 based hadoop distributions that are officially available at hadoop.apache.org - namely 0.23.0, 0.23.1, 2.0.0, 2.0.1. Please note they all are marked as "alpha".

        0.23.0 and 0.23.1 are shipping hsqldb version 1.8.0.7. Versions 2.0.0 and 2.0.1 do not contain any hsqldb jars inside, which make sense as MAPREDUCE-4197 was about adding jars that are requested by examples but not included.

        Considering that:

        • hadoop 0.20 and 1.x is shipping hsqldb 1.8
        • hadoop 2.x is moving hsqldb out
        • other components stayed with 1.8 (oozie is quite important component to us as we're having users actively using it)
        • we've upgraded to 2.x only because hadoop seemed to be also upgrading

        I would personaly incline to revert those two JIRAs. What do you think guys? (I'll not +1 this on Review board and commit it until we will have an agreement)

        Show
        Jarek Jarcec Cecho added a comment - I've took different view on the issue. I've downloaded all 0.23 based hadoop distributions that are officially available at hadoop.apache.org - namely 0.23.0, 0.23.1, 2.0.0, 2.0.1. Please note they all are marked as "alpha". 0.23.0 and 0.23.1 are shipping hsqldb version 1.8.0.7. Versions 2.0.0 and 2.0.1 do not contain any hsqldb jars inside, which make sense as MAPREDUCE-4197 was about adding jars that are requested by examples but not included. Considering that: hadoop 0.20 and 1.x is shipping hsqldb 1.8 hadoop 2.x is moving hsqldb out other components stayed with 1.8 (oozie is quite important component to us as we're having users actively using it) we've upgraded to 2.x only because hadoop seemed to be also upgrading I would personaly incline to revert those two JIRAs. What do you think guys? (I'll not +1 this on Review board and commit it until we will have an agreement)
        Hide
        Abhijeet Gaikwad added a comment -

        As Jarcec suggested, if hsqldb-2.x is not in any hadoop's classpath, then SQOOP-519 is an invalid bug - implies SQOOP-524 is invalid.

        Cheolsoo - Can you please test this patch for a simple import/export with hadoop 2.x and CDH4 clusters, if you have them already. (If you don't have a cluster let me know - can try and setup a single-node.)

        If the tests are a success, I +1 Jarcec's suggestion.

        Show
        Abhijeet Gaikwad added a comment - As Jarcec suggested, if hsqldb-2.x is not in any hadoop's classpath, then SQOOP-519 is an invalid bug - implies SQOOP-524 is invalid. Cheolsoo - Can you please test this patch for a simple import/export with hadoop 2.x and CDH4 clusters, if you have them already. (If you don't have a cluster let me know - can try and setup a single-node.) If the tests are a success, I +1 Jarcec's suggestion.
        Hide
        Jarek Jarcec Cecho added a comment -

        Agreed. I'll wait to see Abhijeet's +1.

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Agreed. I'll wait to see Abhijeet's +1. Jarcec
        Hide
        Cheolsoo Park added a comment -

        Here is what I verified with the hadoop-2.0.0-chd-4.0.1 MR2 cluster.

        1) Upstream sqoop does not have a profile for hadoop-2.0.0, and hadoop-0.23.x is not binary-compatible with hadoop-2.0.0. So to get upstream sqoop working with cdh-4.0.1, I had to modify build.xml so that I can compile it against hadoop-2.0.0-alpha. (I am going to open a jira to add a profile for hadoop-2.0.0 unless there is already one.)

        2) With my patch (i.e. reverting SQOOP-519 and SQOOP0524), I was able to run import/export jobs successfully.

        Thanks!

        Show
        Cheolsoo Park added a comment - Here is what I verified with the hadoop-2.0.0-chd-4.0.1 MR2 cluster. 1) Upstream sqoop does not have a profile for hadoop-2.0.0, and hadoop-0.23.x is not binary-compatible with hadoop-2.0.0. So to get upstream sqoop working with cdh-4.0.1, I had to modify build.xml so that I can compile it against hadoop-2.0.0-alpha. (I am going to open a jira to add a profile for hadoop-2.0.0 unless there is already one.) 2) With my patch (i.e. reverting SQOOP-519 and SQOOP0524), I was able to run import/export jobs successfully. Thanks!
        Hide
        Cheolsoo Park added a comment -

        Filed SQOOP-574 regarding the 2.0.0 build profile.

        Show
        Cheolsoo Park added a comment - Filed SQOOP-574 regarding the 2.0.0 build profile.
        Hide
        Abhijeet Gaikwad added a comment -

        Thanks for the above patch(SQOOP-574) Cheolsoo.

        Verified Apache hadoop 2.x releases do not contain any reference of hsqldb-2.x jars.
        Cheolsoo's experiments sound convincing.

        ant test(20,23,100,200) successful. Will update this on Review board.

        +1

        Thanks.

        Show
        Abhijeet Gaikwad added a comment - Thanks for the above patch( SQOOP-574 ) Cheolsoo. Verified Apache hadoop 2.x releases do not contain any reference of hsqldb-2.x jars. Cheolsoo's experiments sound convincing. ant test(20,23,100,200) successful. Will update this on Review board. +1 Thanks.
        Hide
        Jarek Jarcec Cecho added a comment -

        Patch is in trunk and 1.4.2 branch.

        Thank you for your contribution Cheolsoo!

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Patch is in trunk and 1.4.2 branch. Thank you for your contribution Cheolsoo! Jarcec
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop20 #10 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/10/)
        Changelog update for 1.4.2

        Add:

        (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828)

        Result = SUCCESS
        jarcec :
        Files :

        • /sqoop/trunk/CHANGELOG.txt

        jarcec :
        Files :

        • /sqoop/trunk/bin/configure-sqoop
        • /sqoop/trunk/ivy.xml
        • /sqoop/trunk/ivy/libraries.properties
        • /sqoop/trunk/ivy/sqoop-test.xml
        • /sqoop/trunk/ivy/sqoop.xml
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #10 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/10/ ) Changelog update for 1.4.2 Add: SQOOP-571 Revert SQOOP-519 and SQOOP-524 SQOOP-574 Create a Hadoop-2.0.0 build profile (Revision 1372831) SQOOP-571 . Revert SQOOP-519 and SQOOP-524 . (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828) Result = SUCCESS jarcec : Files : /sqoop/trunk/CHANGELOG.txt jarcec : Files : /sqoop/trunk/bin/configure-sqoop /sqoop/trunk/ivy.xml /sqoop/trunk/ivy/libraries.properties /sqoop/trunk/ivy/sqoop-test.xml /sqoop/trunk/ivy/sqoop.xml /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop23 #140 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/140/)
        Changelog update for 1.4.2

        Add:

        (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828)

        Result = SUCCESS
        jarcec :
        Files :

        • /sqoop/trunk/CHANGELOG.txt

        jarcec :
        Files :

        • /sqoop/trunk/bin/configure-sqoop
        • /sqoop/trunk/ivy.xml
        • /sqoop/trunk/ivy/libraries.properties
        • /sqoop/trunk/ivy/sqoop-test.xml
        • /sqoop/trunk/ivy/sqoop.xml
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #140 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/140/ ) Changelog update for 1.4.2 Add: SQOOP-571 Revert SQOOP-519 and SQOOP-524 SQOOP-574 Create a Hadoop-2.0.0 build profile (Revision 1372831) SQOOP-571 . Revert SQOOP-519 and SQOOP-524 . (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828) Result = SUCCESS jarcec : Files : /sqoop/trunk/CHANGELOG.txt jarcec : Files : /sqoop/trunk/bin/configure-sqoop /sqoop/trunk/ivy.xml /sqoop/trunk/ivy/libraries.properties /sqoop/trunk/ivy/sqoop-test.xml /sqoop/trunk/ivy/sqoop.xml /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop100 #10 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/10/)
        Changelog update for 1.4.2

        Add:

        (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828)

        Result = SUCCESS
        jarcec :
        Files :

        • /sqoop/trunk/CHANGELOG.txt

        jarcec :
        Files :

        • /sqoop/trunk/bin/configure-sqoop
        • /sqoop/trunk/ivy.xml
        • /sqoop/trunk/ivy/libraries.properties
        • /sqoop/trunk/ivy/sqoop-test.xml
        • /sqoop/trunk/ivy/sqoop.xml
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #10 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/10/ ) Changelog update for 1.4.2 Add: SQOOP-571 Revert SQOOP-519 and SQOOP-524 SQOOP-574 Create a Hadoop-2.0.0 build profile (Revision 1372831) SQOOP-571 . Revert SQOOP-519 and SQOOP-524 . (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828) Result = SUCCESS jarcec : Files : /sqoop/trunk/CHANGELOG.txt jarcec : Files : /sqoop/trunk/bin/configure-sqoop /sqoop/trunk/ivy.xml /sqoop/trunk/ivy/libraries.properties /sqoop/trunk/ivy/sqoop-test.xml /sqoop/trunk/ivy/sqoop.xml /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development