Sqoop
  1. Sqoop
  2. SQOOP-482

import tool for mysql direct mode reports wrong record number

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.1-incubating
    • Fix Version/s: 1.4.2
    • Component/s: connectors/mysql
    • Labels:
      None
    • Environment:

      hadoop 1.0.0

      Description

      When run the sqoop import (via mysqldump) task :
      sqoop import --connect jdbc:mysql://[ip]:3306/mysqlslap --table t1 --username root -P --hive-import --hive-overwrite --hive-table tTable --verbose --direct -m 1
      The console shows this(part of the log):
      12/05/02 16:21:16 INFO mapred.JobClient: Map output records=6000594
      12/05/02 16:21:16 INFO mapreduce.ImportJobBase: Retrieved 6000594 records.
      However,the real record number in mysql is 1000099.And the number of records in hive is 1000099 ,too.
      So only the console log report the wrong number.

        Activity

        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #112 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/112/)
        SQOOP-482. Import tool for mysql direct mode reports wrong
        record number
        (Nemon Lou via Kathleen Ting) (Revision 1336434)

        Result = SUCCESS
        kathleen :
        Files :

        • /sqoop/trunk/src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #112 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/112/ ) SQOOP-482 . Import tool for mysql direct mode reports wrong record number (Nemon Lou via Kathleen Ting) (Revision 1336434) Result = SUCCESS kathleen : Files : /sqoop/trunk/src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java
        Hide
        Kathleen Ting added a comment -

        Patch committed. Thanks Nemon!

        Show
        Kathleen Ting added a comment - Patch committed. Thanks Nemon!
        Hide
        Kathleen Ting added a comment -

        Thanks Arvind for adding Nemon to the list.

        Show
        Kathleen Ting added a comment - Thanks Arvind for adding Nemon to the list.
        Hide
        Jarek Jarcec Cecho added a comment -

        Arvind have to put Nemon to Sqoop contributor list before we will be able to assign ticket to him.

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Arvind have to put Nemon to Sqoop contributor list before we will be able to assign ticket to him. Jarcec
        Hide
        Nemon Lou added a comment -

        Sorry.I could not find where to assign this issue to myself.

        Show
        Nemon Lou added a comment - Sorry.I could not find where to assign this issue to myself.
        Hide
        Kathleen Ting added a comment -

        Nemon, please also assign the JIRA to yourself. Thanks.

        Show
        Kathleen Ting added a comment - Nemon, please also assign the JIRA to yourself. Thanks.
        Hide
        Nemon Lou added a comment - - edited

        Update the JIRA patch

        Show
        Nemon Lou added a comment - - edited Update the JIRA patch
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2012-05-04 18:36:37, Cheolsoo Park wrote:

        > Nemon's fix seems to be right to me. The write() method should be called only once per record. I also verified that his patch addresses the problem.

        >

        > @Nemon

        > I am no commiter, so I cannot commit your patch. But you've got my vote!

        @Kate
        To be clear, I am not disagreeing with you. I am just talking about Nemon's diff file on reviewboard, and the patch on the jire has to be uploaded.

        Hope that nobody misinterprets my comment.

        • Cheolsoo

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4988/#review7580
        -----------------------------------------------------------

        On 2012-05-04 07:33:48, nemon lou wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4988/

        -----------------------------------------------------------

        (Updated 2012-05-04 07:33:48)

        Review request for Sqoop, Jarek Cecho and Cheolsoo Park.

        Summary

        -------

        I simply change the way the MySQLDumpMapper writes contents.

        May be there are other ways to fix it.Please let me know,thank you!

        This addresses bug SQOOP-482.

        https://issues.apache.org/jira/browse/SQOOP-482

        Diffs

        -----

        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889

        Diff: https://reviews.apache.org/r/4988/diff

        Testing

        -------

        Thanks,

        nemon

        Show
        jiraposter@reviews.apache.org added a comment - On 2012-05-04 18:36:37, Cheolsoo Park wrote: > Nemon's fix seems to be right to me. The write() method should be called only once per record. I also verified that his patch addresses the problem. > > @Nemon > I am no commiter, so I cannot commit your patch. But you've got my vote! @Kate To be clear, I am not disagreeing with you. I am just talking about Nemon's diff file on reviewboard, and the patch on the jire has to be uploaded. Hope that nobody misinterprets my comment. Cheolsoo ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/#review7580 ----------------------------------------------------------- On 2012-05-04 07:33:48, nemon lou wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/ ----------------------------------------------------------- (Updated 2012-05-04 07:33:48) Review request for Sqoop, Jarek Cecho and Cheolsoo Park. Summary ------- I simply change the way the MySQLDumpMapper writes contents. May be there are other ways to fix it.Please let me know,thank you! This addresses bug SQOOP-482 . https://issues.apache.org/jira/browse/SQOOP-482 Diffs ----- ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889 Diff: https://reviews.apache.org/r/4988/diff Testing ------- Thanks, nemon
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4988/#review7580
        -----------------------------------------------------------

        Ship it!

        Nemon's fix seems to be right to me. The write() method should be called only once per record. I also verified that his patch addresses the problem.

        @Nemon
        I am no commiter, so I cannot commit your patch. But you've got my vote!

        • Cheolsoo

        On 2012-05-04 07:33:48, nemon lou wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4988/

        -----------------------------------------------------------

        (Updated 2012-05-04 07:33:48)

        Review request for Sqoop, Jarek Cecho and Cheolsoo Park.

        Summary

        -------

        I simply change the way the MySQLDumpMapper writes contents.

        May be there are other ways to fix it.Please let me know,thank you!

        This addresses bug SQOOP-482.

        https://issues.apache.org/jira/browse/SQOOP-482

        Diffs

        -----

        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889

        Diff: https://reviews.apache.org/r/4988/diff

        Testing

        -------

        Thanks,

        nemon

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/#review7580 ----------------------------------------------------------- Ship it! Nemon's fix seems to be right to me. The write() method should be called only once per record. I also verified that his patch addresses the problem. @Nemon I am no commiter, so I cannot commit your patch. But you've got my vote! Cheolsoo On 2012-05-04 07:33:48, nemon lou wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/ ----------------------------------------------------------- (Updated 2012-05-04 07:33:48) Review request for Sqoop, Jarek Cecho and Cheolsoo Park. Summary ------- I simply change the way the MySQLDumpMapper writes contents. May be there are other ways to fix it.Please let me know,thank you! This addresses bug SQOOP-482 . https://issues.apache.org/jira/browse/SQOOP-482 Diffs ----- ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889 Diff: https://reviews.apache.org/r/4988/diff Testing ------- Thanks, nemon
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4988/#review7581
        -----------------------------------------------------------

        Ship it!

        +1

        Nemon, thanks for the patch. Noticed that the timestamp of the patch uploaded to the jira is 5/3 but on review board the last diff was made on 5/4. Nemon, would you mind uploading the latest patch to the jira as SQOOP-482-1.patch?

        • Kathleen

        On 2012-05-04 07:33:48, nemon lou wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4988/

        -----------------------------------------------------------

        (Updated 2012-05-04 07:33:48)

        Review request for Sqoop, Jarek Cecho and Cheolsoo Park.

        Summary

        -------

        I simply change the way the MySQLDumpMapper writes contents.

        May be there are other ways to fix it.Please let me know,thank you!

        This addresses bug SQOOP-482.

        https://issues.apache.org/jira/browse/SQOOP-482

        Diffs

        -----

        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889

        Diff: https://reviews.apache.org/r/4988/diff

        Testing

        -------

        Thanks,

        nemon

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/#review7581 ----------------------------------------------------------- Ship it! +1 Nemon, thanks for the patch. Noticed that the timestamp of the patch uploaded to the jira is 5/3 but on review board the last diff was made on 5/4. Nemon, would you mind uploading the latest patch to the jira as SQOOP-482 -1.patch? Kathleen On 2012-05-04 07:33:48, nemon lou wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/ ----------------------------------------------------------- (Updated 2012-05-04 07:33:48) Review request for Sqoop, Jarek Cecho and Cheolsoo Park. Summary ------- I simply change the way the MySQLDumpMapper writes contents. May be there are other ways to fix it.Please let me know,thank you! This addresses bug SQOOP-482 . https://issues.apache.org/jira/browse/SQOOP-482 Diffs ----- ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889 Diff: https://reviews.apache.org/r/4988/diff Testing ------- Thanks, nemon
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4988/
        -----------------------------------------------------------

        (Updated 2012-05-04 07:33:48.277299)

        Review request for Sqoop, Jarek Cecho and Cheolsoo Park.

        Changes
        -------

        checkstyle fixed

        Summary
        -------

        I simply change the way the MySQLDumpMapper writes contents.
        May be there are other ways to fix it.Please let me know,thank you!

        This addresses bug SQOOP-482.
        https://issues.apache.org/jira/browse/SQOOP-482

        Diffs (updated)


        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889

        Diff: https://reviews.apache.org/r/4988/diff

        Testing
        -------

        Thanks,

        nemon

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/ ----------------------------------------------------------- (Updated 2012-05-04 07:33:48.277299) Review request for Sqoop, Jarek Cecho and Cheolsoo Park. Changes ------- checkstyle fixed Summary ------- I simply change the way the MySQLDumpMapper writes contents. May be there are other ways to fix it.Please let me know,thank you! This addresses bug SQOOP-482 . https://issues.apache.org/jira/browse/SQOOP-482 Diffs (updated) ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889 Diff: https://reviews.apache.org/r/4988/diff Testing ------- Thanks, nemon
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4988/#review7546
        -----------------------------------------------------------

        Thank you very much for your patch. It looks good, but there are two minor coding convention issues. If you run "ant checkstyle", you will see the following errors that I commented inline. Would you mind fixing them and updating the patch?

        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java
        <https://reviews.apache.org/r/4988/#comment16716>

        This line is longer than 80 chars.

        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java
        <https://reviews.apache.org/r/4988/#comment16715>

        Please remove a tab char here.

        • Cheolsoo

        On 2012-05-04 01:02:58, nemon lou wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4988/

        -----------------------------------------------------------

        (Updated 2012-05-04 01:02:58)

        Review request for Sqoop, Jarek Cecho and Cheolsoo Park.

        Summary

        -------

        I simply change the way the MySQLDumpMapper writes contents.

        May be there are other ways to fix it.Please let me know,thank you!

        This addresses bug SQOOP-482.

        https://issues.apache.org/jira/browse/SQOOP-482

        Diffs

        -----

        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889

        Diff: https://reviews.apache.org/r/4988/diff

        Testing

        -------

        Thanks,

        nemon

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/#review7546 ----------------------------------------------------------- Thank you very much for your patch. It looks good, but there are two minor coding convention issues. If you run "ant checkstyle", you will see the following errors that I commented inline. Would you mind fixing them and updating the patch? ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java < https://reviews.apache.org/r/4988/#comment16716 > This line is longer than 80 chars. ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java < https://reviews.apache.org/r/4988/#comment16715 > Please remove a tab char here. Cheolsoo On 2012-05-04 01:02:58, nemon lou wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/ ----------------------------------------------------------- (Updated 2012-05-04 01:02:58) Review request for Sqoop, Jarek Cecho and Cheolsoo Park. Summary ------- I simply change the way the MySQLDumpMapper writes contents. May be there are other ways to fix it.Please let me know,thank you! This addresses bug SQOOP-482 . https://issues.apache.org/jira/browse/SQOOP-482 Diffs ----- ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889 Diff: https://reviews.apache.org/r/4988/diff Testing ------- Thanks, nemon
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4988/
        -----------------------------------------------------------

        (Updated 2012-05-04 01:02:58.521635)

        Review request for Sqoop, Jarek Cecho and Cheolsoo Park.

        Changes
        -------

        Add more review people!

        Summary
        -------

        I simply change the way the MySQLDumpMapper writes contents.
        May be there are other ways to fix it.Please let me know,thank you!

        This addresses bug SQOOP-482.
        https://issues.apache.org/jira/browse/SQOOP-482

        Diffs


        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889

        Diff: https://reviews.apache.org/r/4988/diff

        Testing
        -------

        Thanks,

        nemon

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/ ----------------------------------------------------------- (Updated 2012-05-04 01:02:58.521635) Review request for Sqoop, Jarek Cecho and Cheolsoo Park. Changes ------- Add more review people! Summary ------- I simply change the way the MySQLDumpMapper writes contents. May be there are other ways to fix it.Please let me know,thank you! This addresses bug SQOOP-482 . https://issues.apache.org/jira/browse/SQOOP-482 Diffs ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889 Diff: https://reviews.apache.org/r/4988/diff Testing ------- Thanks, nemon
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4988/#review7500
        -----------------------------------------------------------

        • nemon

        On 2012-05-03 10:57:29, nemon lou wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4988/

        -----------------------------------------------------------

        (Updated 2012-05-03 10:57:29)

        Review request for Sqoop and Jarek Cecho.

        Summary

        -------

        I simply change the way the MySQLDumpMapper writes contents.

        May be there are other ways to fix it.Please let me know,thank you!

        This addresses bug SQOOP-482.

        https://issues.apache.org/jira/browse/SQOOP-482

        Diffs

        -----

        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889

        Diff: https://reviews.apache.org/r/4988/diff

        Testing

        -------

        Thanks,

        nemon

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/#review7500 ----------------------------------------------------------- nemon On 2012-05-03 10:57:29, nemon lou wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/ ----------------------------------------------------------- (Updated 2012-05-03 10:57:29) Review request for Sqoop and Jarek Cecho. Summary ------- I simply change the way the MySQLDumpMapper writes contents. May be there are other ways to fix it.Please let me know,thank you! This addresses bug SQOOP-482 . https://issues.apache.org/jira/browse/SQOOP-482 Diffs ----- ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889 Diff: https://reviews.apache.org/r/4988/diff Testing ------- Thanks, nemon
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4988/
        -----------------------------------------------------------

        Review request for Sqoop and Jarek Cecho.

        Summary
        -------

        I simply change the way the MySQLDumpMapper writes contents.
        May be there are other ways to fix it.Please let me know,thank you!

        This addresses bug SQOOP-482.
        https://issues.apache.org/jira/browse/SQOOP-482

        Diffs


        ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889

        Diff: https://reviews.apache.org/r/4988/diff

        Testing
        -------

        Thanks,

        nemon

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4988/ ----------------------------------------------------------- Review request for Sqoop and Jarek Cecho. Summary ------- I simply change the way the MySQLDumpMapper writes contents. May be there are other ways to fix it.Please let me know,thank you! This addresses bug SQOOP-482 . https://issues.apache.org/jira/browse/SQOOP-482 Diffs ./src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java 1332889 Diff: https://reviews.apache.org/r/4988/diff Testing ------- Thanks, nemon
        Hide
        Jarek Jarcec Cecho added a comment -

        Hi Nemon,
        could you please upload your patch to Apache Review board (https://reviews.apache.org/) so that we can review it?

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Hi Nemon, could you please upload your patch to Apache Review board ( https://reviews.apache.org/ ) so that we can review it? Jarcec

          People

          • Assignee:
            Nemon Lou
            Reporter:
            Nemon Lou
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development