Sqoop
  1. Sqoop
  2. SQOOP-519

Update the version of hsqldb decadency to 2.X

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.4.1-incubating
    • Fix Version/s: 1.4.2
    • Component/s: None
    • Labels:
      None

      Description

      Hadoop 2.0.0 depends on hsqldb 2.0.0, and this breaks Sqoop when running against hadoop 2.0.0 since we end up with two different versions of hsqldb in classpath.

      1. SQOOP-519-9.patch
        16 kB
        Cheolsoo Park
      2. SQOOP-519-7.patch
        15 kB
        Cheolsoo Park
      3. SQOOP-519-6.patch
        15 kB
        Cheolsoo Park
      4. SQOOP-519-4.patch
        12 kB
        Cheolsoo Park
      5. SQOOP-519-3.patch
        11 kB
        Cheolsoo Park
      6. SQOOP-519-2.patch
        10 kB
        Cheolsoo Park
      7. SQOOP-519-10.patch
        16 kB
        Cheolsoo Park
      8. SQOOP-519.patch
        3 kB
        Cheolsoo Park

        Activity

        Hide
        Cheolsoo Park added a comment -

        Attached is a patch that updates hsqldb version to 2.0.0.

        Note that I see several test failures after applying this patch and am investigating them.

        Show
        Cheolsoo Park added a comment - Attached is a patch that updates hsqldb version to 2.0.0. Note that I see several test failures after applying this patch and am investigating them.
        Hide
        Cheolsoo Park added a comment -

        This is 2nd patch that fixes the following tests:

        • TestAvroImport
        • TestAvroExport
        • TestColumnTypes
        • TestHiveImport
        • TestMultiCols
        • TestExport
        • TestDataDrivenDBInputFormat

        These tests failed because of

        • data type incompatibility
        • output text change

        Besides, I still see the following tests failing:

        • TestExportUpdate
        • TestIncrementalImport
        • TestSavedJobs

        I will continue to investigate these failures.

        Show
        Cheolsoo Park added a comment - This is 2nd patch that fixes the following tests: TestAvroImport TestAvroExport TestColumnTypes TestHiveImport TestMultiCols TestExport TestDataDrivenDBInputFormat These tests failed because of data type incompatibility output text change Besides, I still see the following tests failing: TestExportUpdate TestIncrementalImport TestSavedJobs I will continue to investigate these failures.
        Hide
        Abhijeet Gaikwad added a comment -

        Can you please give a rough estimate on this, as you know we will start creating SQOOP-1.4.2 release artifacts in around a week.

        ~ Release Manger

        Show
        Abhijeet Gaikwad added a comment - Can you please give a rough estimate on this, as you know we will start creating SQOOP-1.4.2 release artifacts in around a week. ~ Release Manger
        Hide
        Cheolsoo Park added a comment -

        It turned out that Hsqldb 2.0.0 has a bug in terms of handling multi-threaded connections.

        Please use HSQLDB 2.2.8 as there was a problem fixed after 2.2.4 was released (the problem was identical statements belonging to different connections would all get closed if one of them was closed).

        Bumping the version to 2.2.8 TestExportUpdate now passes.

        The remaining failures are now for metastore functionality.

        Show
        Cheolsoo Park added a comment - It turned out that Hsqldb 2.0.0 has a bug in terms of handling multi-threaded connections. Please use HSQLDB 2.2.8 as there was a problem fixed after 2.2.4 was released (the problem was identical statements belonging to different connections would all get closed if one of them was closed). Bumping the version to 2.2.8 TestExportUpdate now passes. The remaining failures are now for metastore functionality.
        Hide
        Cheolsoo Park added a comment -

        The new patch fixes 3rd party unit test. TestIncrementalImport is not fixed.

        Show
        Cheolsoo Park added a comment - The new patch fixes 3rd party unit test. TestIncrementalImport is not fixed.
        Hide
        Cheolsoo Park added a comment -

        The newest patch fixes all unit test cases.

        Regarding TextIncrementalImport, what I end up doing is lowering the level of transaction isolation of Hsqldb from SERIALIZABLE to READ_COMMITTED.

        The problem with using SERIALIZABLE is that it makes the test hang. By looking into JVM thread dump, the test thread hangs at HsqldbJobStorage.java:705. But this is rather strange because it is happening while a single thread makes a transaction to Hsqldb! Nevertheless, it does for whatever reason.

        I don't see any particular issue by lowering the isolation level to READ_COMMITTED (default value), but I suppose that this can change Sqoop metastore's behavior if you set up Sqoop metastore in such a way that multiple users are allowed to access to it simultaneously. Although I haven't heard of such use cases, please let me know if this worries you.

        Thanks!

        Show
        Cheolsoo Park added a comment - The newest patch fixes all unit test cases. Regarding TextIncrementalImport, what I end up doing is lowering the level of transaction isolation of Hsqldb from SERIALIZABLE to READ_COMMITTED. The problem with using SERIALIZABLE is that it makes the test hang. By looking into JVM thread dump, the test thread hangs at HsqldbJobStorage.java:705. But this is rather strange because it is happening while a single thread makes a transaction to Hsqldb! Nevertheless, it does for whatever reason. I don't see any particular issue by lowering the isolation level to READ_COMMITTED (default value), but I suppose that this can change Sqoop metastore's behavior if you set up Sqoop metastore in such a way that multiple users are allowed to access to it simultaneously. Although I haven't heard of such use cases, please let me know if this worries you. Thanks!
        Hide
        Cheolsoo Park added a comment -
        Show
        Cheolsoo Park added a comment - Review board: https://reviews.apache.org/r/6130/
        Hide
        Jarek Jarcec Cecho added a comment -

        I'm working on the review at the moment.

        Show
        Jarek Jarcec Cecho added a comment - I'm working on the review at the moment.
        Hide
        Cheolsoo Park added a comment -

        SQOOP-519-10.patch is the latest.

        Show
        Cheolsoo Park added a comment - SQOOP-519-10.patch is the latest.
        Hide
        Jarek Jarcec Cecho added a comment -

        Committed revision 1365691.

        Thank you for your effort to make this work Cheolsoo!

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Committed revision 1365691. Thank you for your effort to make this work Cheolsoo! Jarcec
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #127 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/127/)
        SQOOP-519. Update the version of hsqldb decadency to 2.X.

        (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1365691)

        Result = SUCCESS
        jarcec :
        Files :

        • /sqoop/trunk/ivy.xml
        • /sqoop/trunk/ivy/libraries.properties
        • /sqoop/trunk/ivy/sqoop-test.xml
        • /sqoop/trunk/ivy/sqoop.xml
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #127 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/127/ ) SQOOP-519 . Update the version of hsqldb decadency to 2.X. (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1365691) Result = SUCCESS jarcec : Files : /sqoop/trunk/ivy.xml /sqoop/trunk/ivy/libraries.properties /sqoop/trunk/ivy/sqoop-test.xml /sqoop/trunk/ivy/sqoop.xml /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop20 #10 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/10/)
        Changelog update for 1.4.2

        Add:

        (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828)

        Result = SUCCESS
        jarcec :
        Files :

        • /sqoop/trunk/CHANGELOG.txt

        jarcec :
        Files :

        • /sqoop/trunk/bin/configure-sqoop
        • /sqoop/trunk/ivy.xml
        • /sqoop/trunk/ivy/libraries.properties
        • /sqoop/trunk/ivy/sqoop-test.xml
        • /sqoop/trunk/ivy/sqoop.xml
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #10 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/10/ ) Changelog update for 1.4.2 Add: SQOOP-571 Revert SQOOP-519 and SQOOP-524 SQOOP-574 Create a Hadoop-2.0.0 build profile (Revision 1372831) SQOOP-571 . Revert SQOOP-519 and SQOOP-524 . (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828) Result = SUCCESS jarcec : Files : /sqoop/trunk/CHANGELOG.txt jarcec : Files : /sqoop/trunk/bin/configure-sqoop /sqoop/trunk/ivy.xml /sqoop/trunk/ivy/libraries.properties /sqoop/trunk/ivy/sqoop-test.xml /sqoop/trunk/ivy/sqoop.xml /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop23 #140 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/140/)
        Changelog update for 1.4.2

        Add:

        (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828)

        Result = SUCCESS
        jarcec :
        Files :

        • /sqoop/trunk/CHANGELOG.txt

        jarcec :
        Files :

        • /sqoop/trunk/bin/configure-sqoop
        • /sqoop/trunk/ivy.xml
        • /sqoop/trunk/ivy/libraries.properties
        • /sqoop/trunk/ivy/sqoop-test.xml
        • /sqoop/trunk/ivy/sqoop.xml
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #140 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/140/ ) Changelog update for 1.4.2 Add: SQOOP-571 Revert SQOOP-519 and SQOOP-524 SQOOP-574 Create a Hadoop-2.0.0 build profile (Revision 1372831) SQOOP-571 . Revert SQOOP-519 and SQOOP-524 . (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828) Result = SUCCESS jarcec : Files : /sqoop/trunk/CHANGELOG.txt jarcec : Files : /sqoop/trunk/bin/configure-sqoop /sqoop/trunk/ivy.xml /sqoop/trunk/ivy/libraries.properties /sqoop/trunk/ivy/sqoop-test.xml /sqoop/trunk/ivy/sqoop.xml /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop100 #10 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/10/)
        Changelog update for 1.4.2

        Add:

        (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828)

        Result = SUCCESS
        jarcec :
        Files :

        • /sqoop/trunk/CHANGELOG.txt

        jarcec :
        Files :

        • /sqoop/trunk/bin/configure-sqoop
        • /sqoop/trunk/ivy.xml
        • /sqoop/trunk/ivy/libraries.properties
        • /sqoop/trunk/ivy/sqoop-test.xml
        • /sqoop/trunk/ivy/sqoop.xml
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java
        • /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #10 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/10/ ) Changelog update for 1.4.2 Add: SQOOP-571 Revert SQOOP-519 and SQOOP-524 SQOOP-574 Create a Hadoop-2.0.0 build profile (Revision 1372831) SQOOP-571 . Revert SQOOP-519 and SQOOP-524 . (Cheolsoo Park via Jarek Jarcec Cecho) (Revision 1372828) Result = SUCCESS jarcec : Files : /sqoop/trunk/CHANGELOG.txt jarcec : Files : /sqoop/trunk/bin/configure-sqoop /sqoop/trunk/ivy.xml /sqoop/trunk/ivy/libraries.properties /sqoop/trunk/ivy/sqoop-test.xml /sqoop/trunk/ivy/sqoop.xml /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbJobStorage.java /sqoop/trunk/src/java/org/apache/sqoop/metastore/hsqldb/HsqldbMetaStore.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestExport.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestMultiCols.java /sqoop/trunk/src/test/com/cloudera/sqoop/hive/TestHiveImport.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleCompatTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/OracleExportTest.java /sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development