Sqoop
  1. Sqoop
  2. SQOOP-409

hive-drop-import-delims doesn't work with MySQL direct import

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0-incubating
    • Fix Version/s: 1.4.1-incubating
    • Component/s: hive-integration
    • Labels:
      None

      Description

      When using hive-drop-import-delims together with direct on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.

      I tried looking at the code but it wasn't easy to follow.

      There seem to be at least two problems:

      • The logic in MySQLDumpMapper.map only looks for MySQLUtils.outputDelimsAreMySQL(conf) to decide if it needs to be reparsed or not, this should also include the hive options
      • This is just a guess at the real problem: The ReparsingStreamThread calls a RecordParser directly without using the created ORM classes and thus circumventing the calls to hiveStringDropDelims
      1. SQOOP-409.patch
        1 kB
        Jarek Jarcec Cecho
      2. SQOOP-409.patch
        1 kB
        Jarek Jarcec Cecho

        Activity

        Hide
        Jarek Jarcec Cecho added a comment -

        Hi Lars,
        this is actually general problem of the direct connectors that are "kind of" different than normal connectors. We're going to fix this discrepancy between different connectors in Sqoop 2. Please try to avoid using --direct option in case that you need to drop hive delimiters if possible for Sqoop 1.x.

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Hi Lars, this is actually general problem of the direct connectors that are "kind of" different than normal connectors. We're going to fix this discrepancy between different connectors in Sqoop 2. Please try to avoid using --direct option in case that you need to drop hive delimiters if possible for Sqoop 1.x. Jarcec
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3506/
        -----------------------------------------------------------

        Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee.

        Summary
        -------

        I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.

        This addresses bug SQOOP-409.
        https://issues.apache.org/jira/browse/SQOOP-409

        Diffs


        /src/java/org/apache/sqoop/tool/ImportTool.java 1231506

        Diff: https://reviews.apache.org/r/3506/diff

        Testing
        -------

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3506/ ----------------------------------------------------------- Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee. Summary ------- I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line. This addresses bug SQOOP-409 . https://issues.apache.org/jira/browse/SQOOP-409 Diffs /src/java/org/apache/sqoop/tool/ImportTool.java 1231506 Diff: https://reviews.apache.org/r/3506/diff Testing ------- Thanks, Jarek
        Hide
        Jarek Jarcec Cecho added a comment -

        I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.

        Show
        Jarek Jarcec Cecho added a comment - I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3506/#review4407
        -----------------------------------------------------------

        Look good overall. Two nits below.

        /src/java/org/apache/sqoop/tool/ImportTool.java
        <https://reviews.apache.org/r/3506/#comment9922>

        Add a space after "if".

        /src/java/org/apache/sqoop/tool/ImportTool.java
        <https://reviews.apache.org/r/3506/#comment9921>

        "+" should be on a new line (from checkstyle).

        • Bilung

        On 2012-01-14 15:09:02, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3506/

        -----------------------------------------------------------

        (Updated 2012-01-14 15:09:02)

        Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee.

        Summary

        -------

        I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.

        This addresses bug SQOOP-409.

        https://issues.apache.org/jira/browse/SQOOP-409

        Diffs

        -----

        /src/java/org/apache/sqoop/tool/ImportTool.java 1231506

        Diff: https://reviews.apache.org/r/3506/diff

        Testing

        -------

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3506/#review4407 ----------------------------------------------------------- Look good overall. Two nits below. /src/java/org/apache/sqoop/tool/ImportTool.java < https://reviews.apache.org/r/3506/#comment9922 > Add a space after "if". /src/java/org/apache/sqoop/tool/ImportTool.java < https://reviews.apache.org/r/3506/#comment9921 > "+" should be on a new line (from checkstyle). Bilung On 2012-01-14 15:09:02, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3506/ ----------------------------------------------------------- (Updated 2012-01-14 15:09:02) Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee. Summary ------- I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line. This addresses bug SQOOP-409 . https://issues.apache.org/jira/browse/SQOOP-409 Diffs ----- /src/java/org/apache/sqoop/tool/ImportTool.java 1231506 Diff: https://reviews.apache.org/r/3506/diff Testing ------- Thanks, Jarek
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3506/
        -----------------------------------------------------------

        (Updated 2012-01-17 18:59:07.118101)

        Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee.

        Changes
        -------

        Thank you for the comments sir, I've completely missed check style step before uploading this patch Sorry about that.

        Summary
        -------

        I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.

        This addresses bug SQOOP-409.
        https://issues.apache.org/jira/browse/SQOOP-409

        Diffs (updated)


        /src/java/org/apache/sqoop/tool/ImportTool.java 1231506

        Diff: https://reviews.apache.org/r/3506/diff

        Testing
        -------

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3506/ ----------------------------------------------------------- (Updated 2012-01-17 18:59:07.118101) Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee. Changes ------- Thank you for the comments sir, I've completely missed check style step before uploading this patch Sorry about that. Summary ------- I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line. This addresses bug SQOOP-409 . https://issues.apache.org/jira/browse/SQOOP-409 Diffs (updated) /src/java/org/apache/sqoop/tool/ImportTool.java 1231506 Diff: https://reviews.apache.org/r/3506/diff Testing ------- Thanks, Jarek
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3506/#review4431
        -----------------------------------------------------------

        Ship it!

        • Bilung

        On 2012-01-17 18:59:07, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3506/

        -----------------------------------------------------------

        (Updated 2012-01-17 18:59:07)

        Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee.

        Summary

        -------

        I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.

        This addresses bug SQOOP-409.

        https://issues.apache.org/jira/browse/SQOOP-409

        Diffs

        -----

        /src/java/org/apache/sqoop/tool/ImportTool.java 1231506

        Diff: https://reviews.apache.org/r/3506/diff

        Testing

        -------

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3506/#review4431 ----------------------------------------------------------- Ship it! Bilung On 2012-01-17 18:59:07, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3506/ ----------------------------------------------------------- (Updated 2012-01-17 18:59:07) Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee. Summary ------- I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line. This addresses bug SQOOP-409 . https://issues.apache.org/jira/browse/SQOOP-409 Diffs ----- /src/java/org/apache/sqoop/tool/ImportTool.java 1231506 Diff: https://reviews.apache.org/r/3506/diff Testing ------- Thanks, Jarek
        Hide
        Bilung Lee added a comment -

        Thanks, Jarcec! Patch committed.

        Show
        Bilung Lee added a comment - Thanks, Jarcec! Patch committed.
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #78 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/78/)
        SQOOP-409 hive-drop-import-delims doesn't work with MySQL direct import

        blee : http://svn.apache.org/viewvc/?view=rev&rev=1232653
        Files :

        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/ImportTool.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #78 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/78/ ) SQOOP-409 hive-drop-import-delims doesn't work with MySQL direct import blee : http://svn.apache.org/viewvc/?view=rev&rev=1232653 Files : /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/ImportTool.java

          People

          • Assignee:
            Jarek Jarcec Cecho
            Reporter:
            Lars Francke
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development