Sqoop
  1. Sqoop
  2. SQOOP-450

Direct MySQL import can fail when using mysqldump version 5.1.14 or above.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.1-incubating
    • Fix Version/s: 1.4.2
    • Component/s: connectors/mysql
    • Labels:
      None

      Description

      When using direct MySQL connector, the expected output of the mysqldump command is individual records as follows:

      INSERT INTO `EMPLOYEES_MYSQL_1` VALUES (2,'Bob',100,'2009-04-20',400,'sales');
      INSERT INTO `EMPLOYEES_MYSQL_1` VALUES (3,'Fred',4000000000,'2009-01-23',15,'marketing');
      

      However, if the client version of MySQL being used is 5.1.14 or above, the output may have a warning header as follows if the tablespaces are not defined:

      --
      -- Not dumping tablespaces as no INFORMATION_SCHEMA.FILES table on this server
      --
      INSERT INTO `EMPLOYEES_MYSQL_1` VALUES (2,'Bob',100,'2009-04-20',400,'sales');
      INSERT INTO `EMPLOYEES_MYSQL_1` VALUES (3,'Fred',4000000000,'2009-01-23',15,'marketing');
      

      Notice that it contains an empty line and other lines that begin with the comment marker --.

      This causes the MySQLDumpMapper to error out due to unexpected format of the records being read.

      1. SQOOP-450-1.patch
        1 kB
        Arvind Prabhakar

        Activity

        Hide
        Arvind Prabhakar added a comment -
        Show
        Arvind Prabhakar added a comment - Relevant: http://bugs.mysql.com/bug.php?id=53728
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4121/
        -----------------------------------------------------------

        Review request for Sqoop.

        Summary
        -------

        See Jira description for problem summary. The fix was to add the logic in the mapper sinks to parse out and ignore empty lines and those that begin with a comment marker.

        This addresses bug SQOOP-450.
        https://issues.apache.org/jira/browse/SQOOP-450

        Diffs


        /src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1295369

        Diff: https://reviews.apache.org/r/4121/diff

        Testing
        -------

        Ran DirectMySQLTest and MySQLAuthTest using mysqldump version 5.5.19 going against server version 5.1.61 where this problem is reproducible. Also tested against a system with MySQL version 5.5.20 where this problem was originally not reproducible.

        Apart from this, ran the customary unit tests and third party tests.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4121/ ----------------------------------------------------------- Review request for Sqoop. Summary ------- See Jira description for problem summary. The fix was to add the logic in the mapper sinks to parse out and ignore empty lines and those that begin with a comment marker. This addresses bug SQOOP-450 . https://issues.apache.org/jira/browse/SQOOP-450 Diffs /src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1295369 Diff: https://reviews.apache.org/r/4121/diff Testing ------- Ran DirectMySQLTest and MySQLAuthTest using mysqldump version 5.5.19 going against server version 5.1.61 where this problem is reproducible. Also tested against a system with MySQL version 5.5.20 where this problem was originally not reproducible. Apart from this, ran the customary unit tests and third party tests. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4121/#review5491
        -----------------------------------------------------------

        Ship it!

        Looks good! Thanks Arvind.

        • Kate

        On 2012-03-01 04:39:42, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4121/

        -----------------------------------------------------------

        (Updated 2012-03-01 04:39:42)

        Review request for Sqoop.

        Summary

        -------

        See Jira description for problem summary. The fix was to add the logic in the mapper sinks to parse out and ignore empty lines and those that begin with a comment marker.

        This addresses bug SQOOP-450.

        https://issues.apache.org/jira/browse/SQOOP-450

        Diffs

        -----

        /src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1295369

        Diff: https://reviews.apache.org/r/4121/diff

        Testing

        -------

        Ran DirectMySQLTest and MySQLAuthTest using mysqldump version 5.5.19 going against server version 5.1.61 where this problem is reproducible. Also tested against a system with MySQL version 5.5.20 where this problem was originally not reproducible.

        Apart from this, ran the customary unit tests and third party tests.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4121/#review5491 ----------------------------------------------------------- Ship it! Looks good! Thanks Arvind. Kate On 2012-03-01 04:39:42, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4121/ ----------------------------------------------------------- (Updated 2012-03-01 04:39:42) Review request for Sqoop. Summary ------- See Jira description for problem summary. The fix was to add the logic in the mapper sinks to parse out and ignore empty lines and those that begin with a comment marker. This addresses bug SQOOP-450 . https://issues.apache.org/jira/browse/SQOOP-450 Diffs ----- /src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1295369 Diff: https://reviews.apache.org/r/4121/diff Testing ------- Ran DirectMySQLTest and MySQLAuthTest using mysqldump version 5.5.19 going against server version 5.1.61 where this problem is reproducible. Also tested against a system with MySQL version 5.5.20 where this problem was originally not reproducible. Apart from this, ran the customary unit tests and third party tests. Thanks, Arvind
        Hide
        Kathleen Ting added a comment -

        Patch committed. Thanks Arvind!

        Show
        Kathleen Ting added a comment - Patch committed. Thanks Arvind!
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #91 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/91/)
        SQOOP-450. Direct MySQL import can fail when using mysqldump version 5.1.14 or above.

        (Arvind Prabhakar via Kathleen Ting) (Revision 1295750)

        Result = SUCCESS
        kathleen : http://svn.apache.org/viewvc/?view=rev&rev=1295750
        Files :

        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #91 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/91/ ) SQOOP-450 . Direct MySQL import can fail when using mysqldump version 5.1.14 or above. (Arvind Prabhakar via Kathleen Ting) (Revision 1295750) Result = SUCCESS kathleen : http://svn.apache.org/viewvc/?view=rev&rev=1295750 Files : /incubator/sqoop/trunk/src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java

          People

          • Assignee:
            Arvind Prabhakar
            Reporter:
            Arvind Prabhakar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development