Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-450

Direct MySQL import can fail when using mysqldump version 5.1.14 or above.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.1-incubating
    • Fix Version/s: 1.4.2
    • Component/s: connectors/mysql
    • Labels:
      None

      Description

      When using direct MySQL connector, the expected output of the mysqldump command is individual records as follows:

      INSERT INTO `EMPLOYEES_MYSQL_1` VALUES (2,'Bob',100,'2009-04-20',400,'sales');
      INSERT INTO `EMPLOYEES_MYSQL_1` VALUES (3,'Fred',4000000000,'2009-01-23',15,'marketing');
      

      However, if the client version of MySQL being used is 5.1.14 or above, the output may have a warning header as follows if the tablespaces are not defined:

      --
      -- Not dumping tablespaces as no INFORMATION_SCHEMA.FILES table on this server
      --
      INSERT INTO `EMPLOYEES_MYSQL_1` VALUES (2,'Bob',100,'2009-04-20',400,'sales');
      INSERT INTO `EMPLOYEES_MYSQL_1` VALUES (3,'Fred',4000000000,'2009-01-23',15,'marketing');
      

      Notice that it contains an empty line and other lines that begin with the comment marker --.

      This causes the MySQLDumpMapper to error out due to unexpected format of the records being read.

      1. SQOOP-450-1.patch
        1 kB
        Arvind Prabhakar

        Activity

        Hide
        aprabhakar Arvind Prabhakar added a comment -
        Show
        aprabhakar Arvind Prabhakar added a comment - Relevant: http://bugs.mysql.com/bug.php?id=53728
        Hide
        jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4121/
        -----------------------------------------------------------

        Review request for Sqoop.

        Summary
        -------

        See Jira description for problem summary. The fix was to add the logic in the mapper sinks to parse out and ignore empty lines and those that begin with a comment marker.

        This addresses bug SQOOP-450.
        https://issues.apache.org/jira/browse/SQOOP-450

        Diffs


        /src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1295369

        Diff: https://reviews.apache.org/r/4121/diff

        Testing
        -------

        Ran DirectMySQLTest and MySQLAuthTest using mysqldump version 5.5.19 going against server version 5.1.61 where this problem is reproducible. Also tested against a system with MySQL version 5.5.20 where this problem was originally not reproducible.

        Apart from this, ran the customary unit tests and third party tests.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4121/ ----------------------------------------------------------- Review request for Sqoop. Summary ------- See Jira description for problem summary. The fix was to add the logic in the mapper sinks to parse out and ignore empty lines and those that begin with a comment marker. This addresses bug SQOOP-450 . https://issues.apache.org/jira/browse/SQOOP-450 Diffs /src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1295369 Diff: https://reviews.apache.org/r/4121/diff Testing ------- Ran DirectMySQLTest and MySQLAuthTest using mysqldump version 5.5.19 going against server version 5.1.61 where this problem is reproducible. Also tested against a system with MySQL version 5.5.20 where this problem was originally not reproducible. Apart from this, ran the customary unit tests and third party tests. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4121/#review5491
        -----------------------------------------------------------

        Ship it!

        Looks good! Thanks Arvind.

        • Kate

        On 2012-03-01 04:39:42, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4121/

        -----------------------------------------------------------

        (Updated 2012-03-01 04:39:42)

        Review request for Sqoop.

        Summary

        -------

        See Jira description for problem summary. The fix was to add the logic in the mapper sinks to parse out and ignore empty lines and those that begin with a comment marker.

        This addresses bug SQOOP-450.

        https://issues.apache.org/jira/browse/SQOOP-450

        Diffs

        -----

        /src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1295369

        Diff: https://reviews.apache.org/r/4121/diff

        Testing

        -------

        Ran DirectMySQLTest and MySQLAuthTest using mysqldump version 5.5.19 going against server version 5.1.61 where this problem is reproducible. Also tested against a system with MySQL version 5.5.20 where this problem was originally not reproducible.

        Apart from this, ran the customary unit tests and third party tests.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4121/#review5491 ----------------------------------------------------------- Ship it! Looks good! Thanks Arvind. Kate On 2012-03-01 04:39:42, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4121/ ----------------------------------------------------------- (Updated 2012-03-01 04:39:42) Review request for Sqoop. Summary ------- See Jira description for problem summary. The fix was to add the logic in the mapper sinks to parse out and ignore empty lines and those that begin with a comment marker. This addresses bug SQOOP-450 . https://issues.apache.org/jira/browse/SQOOP-450 Diffs ----- /src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1295369 Diff: https://reviews.apache.org/r/4121/diff Testing ------- Ran DirectMySQLTest and MySQLAuthTest using mysqldump version 5.5.19 going against server version 5.1.61 where this problem is reproducible. Also tested against a system with MySQL version 5.5.20 where this problem was originally not reproducible. Apart from this, ran the customary unit tests and third party tests. Thanks, Arvind
        Hide
        kathleen Kathleen Ting added a comment -

        Patch committed. Thanks Arvind!

        Show
        kathleen Kathleen Ting added a comment - Patch committed. Thanks Arvind!
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #91 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/91/)
        SQOOP-450. Direct MySQL import can fail when using mysqldump version 5.1.14 or above.

        (Arvind Prabhakar via Kathleen Ting) (Revision 1295750)

        Result = SUCCESS
        kathleen : http://svn.apache.org/viewvc/?view=rev&rev=1295750
        Files :

        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #91 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/91/ ) SQOOP-450 . Direct MySQL import can fail when using mysqldump version 5.1.14 or above. (Arvind Prabhakar via Kathleen Ting) (Revision 1295750) Result = SUCCESS kathleen : http://svn.apache.org/viewvc/?view=rev&rev=1295750 Files : /incubator/sqoop/trunk/src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java

          People

          • Assignee:
            aprabhakar Arvind Prabhakar
            Reporter:
            aprabhakar Arvind Prabhakar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development