Sqoop
  1. Sqoop
  2. SQOOP-473

Sqoop import-all-tables fails with MySQL connector

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.1-incubating
    • Fix Version/s: 1.4.2
    • Component/s: connectors
    • Labels:
      None

      Description

      Sqoop command
      sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables

      Output:

      12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
      12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
      12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
      12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
      java.lang.NullPointerException
      	at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
      	at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
      	at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
      	at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
      	at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
      	at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
      	at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
      	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
      	at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
      	at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
      	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
      	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
      	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
      	at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
      	at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
      
      

      I have not tested with other connectors.

      1. SQOOP-473.patch
        12 kB
        Cheolsoo Park

        Activity

        Hide
        Cheolsoo Park added a comment - - edited

        Bilung suggested the following fix for this problem, and I like his suggestion.

        For all-tables import, the table name is not specified, so in the getColumnTypeNames() method of ConnManager, the 'else' path is taken in the following code:

        if (null != tableName) {
           // We're generating a class based on a table import.
           columnTypeNames = getColumnTypeNamesForTable(tableName);
        } else {
           // This is based on an arbitrary query.
           String query = sqlQuery;
           if (query.indexOf(SqlManager.SUBSTITUTE_TOKEN) == -1) {
             throw new RuntimeException("Query [" + query + "] must contain '"
                 + SqlManager.SUBSTITUTE_TOKEN + "' in WHERE clause.");
           }
           columnTypeNames = getColumnTypeNamesForQuery(query);
        }
        

        However, this is wrong since all-tables import is not free form query import.

        To fix the problem, I am going to modify the signature of the toJavaType() method so that tableName can be given as an argument in addition to columnName and sqlType and passed down to getColumnTypeNames(). Since tableName is available at all the places where toJavaType() is called, only minor changes are needed.

        In addition, I am going to modify the toHiveType() and toAvroType() methods so that the signature of these methods can stay consistent with that of toJavaType() (although this is not required).

        Show
        Cheolsoo Park added a comment - - edited Bilung suggested the following fix for this problem, and I like his suggestion. For all-tables import, the table name is not specified, so in the getColumnTypeNames() method of ConnManager, the 'else' path is taken in the following code: if ( null != tableName) { // We're generating a class based on a table import . columnTypeNames = getColumnTypeNamesForTable(tableName); } else { // This is based on an arbitrary query. String query = sqlQuery; if (query.indexOf(SqlManager.SUBSTITUTE_TOKEN) == -1) { throw new RuntimeException( "Query [" + query + "] must contain '" + SqlManager.SUBSTITUTE_TOKEN + "' in WHERE clause." ); } columnTypeNames = getColumnTypeNamesForQuery(query); } However, this is wrong since all-tables import is not free form query import. To fix the problem, I am going to modify the signature of the toJavaType() method so that tableName can be given as an argument in addition to columnName and sqlType and passed down to getColumnTypeNames(). Since tableName is available at all the places where toJavaType() is called, only minor changes are needed. In addition, I am going to modify the toHiveType() and toAvroType() methods so that the signature of these methods can stay consistent with that of toJavaType() (although this is not required).
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4584/
        -----------------------------------------------------------

        Review request for Sqoop.

        Summary
        -------

        All-tables import fails with MySQL:

        https://issues.apache.org/jira/browse/SQOOP-473

        The change include:

        1) Modify toJavaType(), toHiveType(), and toAvroType() methods.
        2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).
        3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.

        This addresses bug SQOOP-473.
        https://issues.apache.org/jira/browse/SQOOP-473

        Diffs


        src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391
        src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82
        src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c
        src/java/org/apache/sqoop/manager/OracleManager.java 30401c5
        src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834
        src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801
        src/java/org/apache/sqoop/orm/ClassWriter.java 986934d
        src/test/com/cloudera/sqoop/TestAllTables.java e6561e9
        src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29

        Diff: https://reviews.apache.org/r/4584/diff

        Testing
        -------

        1) Verified that MySQL all-tables import is fixed using new unit test.
        2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.

        Thanks,

        Cheolsoo

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4584/ ----------------------------------------------------------- Review request for Sqoop. Summary ------- All-tables import fails with MySQL: https://issues.apache.org/jira/browse/SQOOP-473 The change include: 1) Modify toJavaType(), toHiveType(), and toAvroType() methods. 2) Add new unit test for MySQL all-tables import (subclassing TestAllTables). 3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers. This addresses bug SQOOP-473 . https://issues.apache.org/jira/browse/SQOOP-473 Diffs src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391 src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82 src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c src/java/org/apache/sqoop/manager/OracleManager.java 30401c5 src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834 src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801 src/java/org/apache/sqoop/orm/ClassWriter.java 986934d src/test/com/cloudera/sqoop/TestAllTables.java e6561e9 src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29 Diff: https://reviews.apache.org/r/4584/diff Testing ------- 1) Verified that MySQL all-tables import is fixed using new unit test. 2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle. Thanks, Cheolsoo
        Hide
        Cheolsoo Park added a comment -

        While working on my patch, I also noticed that all-tables import does not work with Oracle at all. But I am not going to address Oracle in this Jira because it is a totally different issue. According to a test that I quickly wrote, it fails because the query that is executed to obtain all tables in the db returns more tables than what the user has actually created. A new Jira will be opened to track this issue.

        Here is a call stack that I get from my test with Oracle for whom may be curious:

        java.lang.IllegalArgumentException: Attempted to generate class with no columns!
            at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1251)
            at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1139)
            at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
            at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
            at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
            at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
            at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
            at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:45)
            at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:215)
            at com.cloudera.sqoop.TestAllTables.testMultiTableImport(TestAllTables.java:106)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:616)
            at junit.framework.TestCase.runTest(TestCase.java:168)
            at junit.framework.TestCase.runBare(TestCase.java:134)
            at junit.framework.TestResult$1.protect(TestResult.java:110)
            at junit.framework.TestResult.runProtected(TestResult.java:128)
            at junit.framework.TestResult.run(TestResult.java:113)
            at junit.framework.TestCase.run(TestCase.java:124)
            at junit.framework.TestSuite.runTest(TestSuite.java:243)
            at junit.framework.TestSuite.run(TestSuite.java:238)
            at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
            at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
            at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
            at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
            at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
        
        Show
        Cheolsoo Park added a comment - While working on my patch, I also noticed that all-tables import does not work with Oracle at all. But I am not going to address Oracle in this Jira because it is a totally different issue. According to a test that I quickly wrote, it fails because the query that is executed to obtain all tables in the db returns more tables than what the user has actually created. A new Jira will be opened to track this issue. Here is a call stack that I get from my test with Oracle for whom may be curious: java.lang.IllegalArgumentException: Attempted to generate class with no columns! at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1251) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1139) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390) at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64) at org.apache.sqoop.Sqoop.run(Sqoop.java:145) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:45) at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:215) at com.cloudera.sqoop.TestAllTables.testMultiTableImport(TestAllTables.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4584/
        -----------------------------------------------------------

        (Updated 2012-04-02 17:50:28.671860)

        Review request for Sqoop.

        Summary
        -------

        All-tables import fails with MySQL:

        https://issues.apache.org/jira/browse/SQOOP-473

        The change include:

        1) Modify toJavaType(), toHiveType(), and toAvroType() methods.
        2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).
        3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.

        This addresses bug SQOOP-473.
        https://issues.apache.org/jira/browse/SQOOP-473

        Diffs (updated)


        src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391
        src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82
        src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c
        src/java/org/apache/sqoop/manager/OracleManager.java 30401c5
        src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834
        src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801
        src/java/org/apache/sqoop/orm/ClassWriter.java 986934d
        src/test/com/cloudera/sqoop/TestAllTables.java e6561e9
        src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29

        Diff: https://reviews.apache.org/r/4584/diff

        Testing
        -------

        1) Verified that MySQL all-tables import is fixed using new unit test.
        2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.

        Thanks,

        Cheolsoo

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4584/ ----------------------------------------------------------- (Updated 2012-04-02 17:50:28.671860) Review request for Sqoop. Summary ------- All-tables import fails with MySQL: https://issues.apache.org/jira/browse/SQOOP-473 The change include: 1) Modify toJavaType(), toHiveType(), and toAvroType() methods. 2) Add new unit test for MySQL all-tables import (subclassing TestAllTables). 3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers. This addresses bug SQOOP-473 . https://issues.apache.org/jira/browse/SQOOP-473 Diffs (updated) src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391 src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82 src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c src/java/org/apache/sqoop/manager/OracleManager.java 30401c5 src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834 src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801 src/java/org/apache/sqoop/orm/ClassWriter.java 986934d src/test/com/cloudera/sqoop/TestAllTables.java e6561e9 src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29 Diff: https://reviews.apache.org/r/4584/diff Testing ------- 1) Verified that MySQL all-tables import is fixed using new unit test. 2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle. Thanks, Cheolsoo
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4584/#review6650
        -----------------------------------------------------------

        Thanks for the patch. Some comments below.

        src/java/org/apache/sqoop/manager/MySQLManager.java
        <https://reviews.apache.org/r/4584/#comment14346>

        Also need to use the tableName parameter here.

        src/java/org/apache/sqoop/manager/MySQLManager.java
        <https://reviews.apache.org/r/4584/#comment14347>

        Ditto.

        • Bilung

        On 2012-04-02 17:50:28, Cheolsoo Park wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4584/

        -----------------------------------------------------------

        (Updated 2012-04-02 17:50:28)

        Review request for Sqoop.

        Summary

        -------

        All-tables import fails with MySQL:

        https://issues.apache.org/jira/browse/SQOOP-473

        The change include:

        1) Modify toJavaType(), toHiveType(), and toAvroType() methods.

        2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).

        3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.

        This addresses bug SQOOP-473.

        https://issues.apache.org/jira/browse/SQOOP-473

        Diffs

        -----

        src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391

        src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82

        src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c

        src/java/org/apache/sqoop/manager/OracleManager.java 30401c5

        src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834

        src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801

        src/java/org/apache/sqoop/orm/ClassWriter.java 986934d

        src/test/com/cloudera/sqoop/TestAllTables.java e6561e9

        src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29

        Diff: https://reviews.apache.org/r/4584/diff

        Testing

        -------

        1) Verified that MySQL all-tables import is fixed using new unit test.

        2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.

        Thanks,

        Cheolsoo

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4584/#review6650 ----------------------------------------------------------- Thanks for the patch. Some comments below. src/java/org/apache/sqoop/manager/MySQLManager.java < https://reviews.apache.org/r/4584/#comment14346 > Also need to use the tableName parameter here. src/java/org/apache/sqoop/manager/MySQLManager.java < https://reviews.apache.org/r/4584/#comment14347 > Ditto. Bilung On 2012-04-02 17:50:28, Cheolsoo Park wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4584/ ----------------------------------------------------------- (Updated 2012-04-02 17:50:28) Review request for Sqoop. Summary ------- All-tables import fails with MySQL: https://issues.apache.org/jira/browse/SQOOP-473 The change include: 1) Modify toJavaType(), toHiveType(), and toAvroType() methods. 2) Add new unit test for MySQL all-tables import (subclassing TestAllTables). 3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers. This addresses bug SQOOP-473 . https://issues.apache.org/jira/browse/SQOOP-473 Diffs ----- src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391 src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82 src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c src/java/org/apache/sqoop/manager/OracleManager.java 30401c5 src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834 src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801 src/java/org/apache/sqoop/orm/ClassWriter.java 986934d src/test/com/cloudera/sqoop/TestAllTables.java e6561e9 src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29 Diff: https://reviews.apache.org/r/4584/diff Testing ------- 1) Verified that MySQL all-tables import is fixed using new unit test. 2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle. Thanks, Cheolsoo
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4584/
        -----------------------------------------------------------

        (Updated 2012-04-03 19:13:20.381070)

        Review request for Sqoop.

        Changes
        -------

        1) Incorporate Bilung's comments in toHiveType() and to AvroType().
        2) Remove a unnecessary method from MySQLAllTablesTest.

        Summary
        -------

        All-tables import fails with MySQL:

        https://issues.apache.org/jira/browse/SQOOP-473

        The change include:

        1) Modify toJavaType(), toHiveType(), and toAvroType() methods.
        2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).
        3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.

        This addresses bug SQOOP-473.
        https://issues.apache.org/jira/browse/SQOOP-473

        Diffs (updated)


        src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391
        src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82
        src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c
        src/java/org/apache/sqoop/manager/OracleManager.java 30401c5
        src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834
        src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801
        src/java/org/apache/sqoop/orm/ClassWriter.java 986934d
        src/test/com/cloudera/sqoop/TestAllTables.java e6561e9
        src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29

        Diff: https://reviews.apache.org/r/4584/diff

        Testing
        -------

        1) Verified that MySQL all-tables import is fixed using new unit test.
        2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.

        Thanks,

        Cheolsoo

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4584/ ----------------------------------------------------------- (Updated 2012-04-03 19:13:20.381070) Review request for Sqoop. Changes ------- 1) Incorporate Bilung's comments in toHiveType() and to AvroType(). 2) Remove a unnecessary method from MySQLAllTablesTest. Summary ------- All-tables import fails with MySQL: https://issues.apache.org/jira/browse/SQOOP-473 The change include: 1) Modify toJavaType(), toHiveType(), and toAvroType() methods. 2) Add new unit test for MySQL all-tables import (subclassing TestAllTables). 3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers. This addresses bug SQOOP-473 . https://issues.apache.org/jira/browse/SQOOP-473 Diffs (updated) src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391 src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82 src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c src/java/org/apache/sqoop/manager/OracleManager.java 30401c5 src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834 src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801 src/java/org/apache/sqoop/orm/ClassWriter.java 986934d src/test/com/cloudera/sqoop/TestAllTables.java e6561e9 src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29 Diff: https://reviews.apache.org/r/4584/diff Testing ------- 1) Verified that MySQL all-tables import is fixed using new unit test. 2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle. Thanks, Cheolsoo
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4584/#review6665
        -----------------------------------------------------------

        Ship it!

        • Bilung

        On 2012-04-03 19:13:20, Cheolsoo Park wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4584/

        -----------------------------------------------------------

        (Updated 2012-04-03 19:13:20)

        Review request for Sqoop.

        Summary

        -------

        All-tables import fails with MySQL:

        https://issues.apache.org/jira/browse/SQOOP-473

        The change include:

        1) Modify toJavaType(), toHiveType(), and toAvroType() methods.

        2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).

        3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.

        This addresses bug SQOOP-473.

        https://issues.apache.org/jira/browse/SQOOP-473

        Diffs

        -----

        src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391

        src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82

        src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c

        src/java/org/apache/sqoop/manager/OracleManager.java 30401c5

        src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834

        src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801

        src/java/org/apache/sqoop/orm/ClassWriter.java 986934d

        src/test/com/cloudera/sqoop/TestAllTables.java e6561e9

        src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29

        Diff: https://reviews.apache.org/r/4584/diff

        Testing

        -------

        1) Verified that MySQL all-tables import is fixed using new unit test.

        2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.

        Thanks,

        Cheolsoo

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4584/#review6665 ----------------------------------------------------------- Ship it! Bilung On 2012-04-03 19:13:20, Cheolsoo Park wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4584/ ----------------------------------------------------------- (Updated 2012-04-03 19:13:20) Review request for Sqoop. Summary ------- All-tables import fails with MySQL: https://issues.apache.org/jira/browse/SQOOP-473 The change include: 1) Modify toJavaType(), toHiveType(), and toAvroType() methods. 2) Add new unit test for MySQL all-tables import (subclassing TestAllTables). 3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers. This addresses bug SQOOP-473 . https://issues.apache.org/jira/browse/SQOOP-473 Diffs ----- src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391 src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82 src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c src/java/org/apache/sqoop/manager/OracleManager.java 30401c5 src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834 src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801 src/java/org/apache/sqoop/orm/ClassWriter.java 986934d src/test/com/cloudera/sqoop/TestAllTables.java e6561e9 src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29 Diff: https://reviews.apache.org/r/4584/diff Testing ------- 1) Verified that MySQL all-tables import is fixed using new unit test. 2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle. Thanks, Cheolsoo
        Hide
        Bilung Lee added a comment -

        Patch is committed. Thanks!

        Show
        Bilung Lee added a comment - Patch is committed. Thanks!
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #105 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/105/)
        SQOOP-473 Sqoop import-all-tables fails with MySQL connector (Revision 1309163)

        Result = SUCCESS
        blee :
        Files :

        • /sqoop/trunk/src/java/org/apache/sqoop/hive/TableDefWriter.java
        • /sqoop/trunk/src/java/org/apache/sqoop/manager/ConnManager.java
        • /sqoop/trunk/src/java/org/apache/sqoop/manager/MySQLManager.java
        • /sqoop/trunk/src/java/org/apache/sqoop/manager/OracleManager.java
        • /sqoop/trunk/src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
        • /sqoop/trunk/src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
        • /sqoop/trunk/src/java/org/apache/sqoop/orm/ClassWriter.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/TestAllTables.java
        • /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #105 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/105/ ) SQOOP-473 Sqoop import-all-tables fails with MySQL connector (Revision 1309163) Result = SUCCESS blee : Files : /sqoop/trunk/src/java/org/apache/sqoop/hive/TableDefWriter.java /sqoop/trunk/src/java/org/apache/sqoop/manager/ConnManager.java /sqoop/trunk/src/java/org/apache/sqoop/manager/MySQLManager.java /sqoop/trunk/src/java/org/apache/sqoop/manager/OracleManager.java /sqoop/trunk/src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java /sqoop/trunk/src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java /sqoop/trunk/src/java/org/apache/sqoop/orm/ClassWriter.java /sqoop/trunk/src/test/com/cloudera/sqoop/TestAllTables.java /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Prashant Gokhale
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development