Sqoop
  1. Sqoop
  2. SQOOP-444

Support incremental import for free form queries

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.2
    • Component/s: None
    • Labels:
      None

      Description

      Would be nice if sqoop would support incremental import for free form queries.

      1. SQOOP-444.patch
        6 kB
        Jarek Jarcec Cecho
      2. SQOOP-444.patch
        6 kB
        Jarek Jarcec Cecho

        Activity

        Jarek Jarcec Cecho created issue -
        Jarek Jarcec Cecho made changes -
        Field Original Value New Value
        Attachment SQOOP-444.patch [ 12515192 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3961/
        -----------------------------------------------------------

        Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

        Summary
        -------

        I've incorporated support for incremental import of free form queries to sqoop.

        This addresses bug SQOOP-444.
        https://issues.apache.org/jira/browse/SQOOP-444

        Diffs


        /src/java/org/apache/sqoop/tool/ImportTool.java 1245157
        /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1245157

        Diff: https://reviews.apache.org/r/3961/diff

        Testing
        -------

        ant -Dhadoopversion=

        {20,23,100}

        test
        live testing on CDH3 and Microsoft SQL Server

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/ ----------------------------------------------------------- Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- I've incorporated support for incremental import of free form queries to sqoop. This addresses bug SQOOP-444 . https://issues.apache.org/jira/browse/SQOOP-444 Diffs /src/java/org/apache/sqoop/tool/ImportTool.java 1245157 /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1245157 Diff: https://reviews.apache.org/r/3961/diff Testing ------- ant -Dhadoopversion= {20,23,100} test live testing on CDH3 and Microsoft SQL Server Thanks, Jarek
        Jarek Jarcec Cecho made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3961/#review5344
        -----------------------------------------------------------

        Thanks for the patch! Two comments below.

        /src/java/org/apache/sqoop/tool/ImportTool.java
        <https://reviews.apache.org/r/3961/#comment11659>

        This syntax ("AS alias") doesn't seem to work with Oracle. You may want to try dropping the "AS" keyword, which seems to be accepted by most major databases.

        /src/java/org/apache/sqoop/tool/ImportTool.java
        <https://reviews.apache.org/r/3961/#comment11648>

        This replacement is specific to free form query? If so, better move this to the else block above.

        • Bilung

        On 2012-02-19 20:35:48, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3961/

        -----------------------------------------------------------

        (Updated 2012-02-19 20:35:48)

        Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

        Summary

        -------

        I've incorporated support for incremental import of free form queries to sqoop.

        This addresses bug SQOOP-444.

        https://issues.apache.org/jira/browse/SQOOP-444

        Diffs

        -----

        /src/java/org/apache/sqoop/tool/ImportTool.java 1245157

        /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1245157

        Diff: https://reviews.apache.org/r/3961/diff

        Testing

        -------

        ant -Dhadoopversion={20,23,100} test

        live testing on CDH3 and Microsoft SQL Server

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/#review5344 ----------------------------------------------------------- Thanks for the patch! Two comments below. /src/java/org/apache/sqoop/tool/ImportTool.java < https://reviews.apache.org/r/3961/#comment11659 > This syntax ("AS alias") doesn't seem to work with Oracle. You may want to try dropping the "AS" keyword, which seems to be accepted by most major databases. /src/java/org/apache/sqoop/tool/ImportTool.java < https://reviews.apache.org/r/3961/#comment11648 > This replacement is specific to free form query? If so, better move this to the else block above. Bilung On 2012-02-19 20:35:48, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/ ----------------------------------------------------------- (Updated 2012-02-19 20:35:48) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- I've incorporated support for incremental import of free form queries to sqoop. This addresses bug SQOOP-444 . https://issues.apache.org/jira/browse/SQOOP-444 Diffs ----- /src/java/org/apache/sqoop/tool/ImportTool.java 1245157 /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1245157 Diff: https://reviews.apache.org/r/3961/diff Testing ------- ant -Dhadoopversion={20,23,100} test live testing on CDH3 and Microsoft SQL Server Thanks, Jarek
        Jarek Jarcec Cecho made changes -
        Attachment SQOOP-444.patch [ 12516333 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3961/
        -----------------------------------------------------------

        (Updated 2012-02-28 14:43:52.891865)

        Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

        Changes
        -------

        I've applied both suggestions and try it again in live testing environment against Microsoft SQL Server and MySQL.

        Summary
        -------

        I've incorporated support for incremental import of free form queries to sqoop.

        This addresses bug SQOOP-444.
        https://issues.apache.org/jira/browse/SQOOP-444

        Diffs (updated)


        /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1294667
        /src/java/org/apache/sqoop/tool/ImportTool.java 1294667

        Diff: https://reviews.apache.org/r/3961/diff

        Testing
        -------

        ant -Dhadoopversion=

        {20,23,100}

        test
        live testing on CDH3 and Microsoft SQL Server

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/ ----------------------------------------------------------- (Updated 2012-02-28 14:43:52.891865) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Changes ------- I've applied both suggestions and try it again in live testing environment against Microsoft SQL Server and MySQL. Summary ------- I've incorporated support for incremental import of free form queries to sqoop. This addresses bug SQOOP-444 . https://issues.apache.org/jira/browse/SQOOP-444 Diffs (updated) /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1294667 /src/java/org/apache/sqoop/tool/ImportTool.java 1294667 Diff: https://reviews.apache.org/r/3961/diff Testing ------- ant -Dhadoopversion= {20,23,100} test live testing on CDH3 and Microsoft SQL Server Thanks, Jarek
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2012-02-27 20:02:04, Bilung Lee wrote:

        > /src/java/org/apache/sqoop/tool/ImportTool.java, line 183

        > <https://reviews.apache.org/r/3961/diff/1/?file=75912#file75912line183>

        >

        > This syntax ("AS alias") doesn't seem to work with Oracle. You may want to try dropping the "AS" keyword, which seems to be accepted by most major databases.

        Thank you for checking against Oracle sir, I do not have that option at the moment.

        • Jarek

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3961/#review5344
        -----------------------------------------------------------

        On 2012-02-28 14:43:52, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3961/

        -----------------------------------------------------------

        (Updated 2012-02-28 14:43:52)

        Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

        Summary

        -------

        I've incorporated support for incremental import of free form queries to sqoop.

        This addresses bug SQOOP-444.

        https://issues.apache.org/jira/browse/SQOOP-444

        Diffs

        -----

        /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1294667

        /src/java/org/apache/sqoop/tool/ImportTool.java 1294667

        Diff: https://reviews.apache.org/r/3961/diff

        Testing

        -------

        ant -Dhadoopversion={20,23,100} test

        live testing on CDH3 and Microsoft SQL Server

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - On 2012-02-27 20:02:04, Bilung Lee wrote: > /src/java/org/apache/sqoop/tool/ImportTool.java, line 183 > < https://reviews.apache.org/r/3961/diff/1/?file=75912#file75912line183 > > > This syntax ("AS alias") doesn't seem to work with Oracle. You may want to try dropping the "AS" keyword, which seems to be accepted by most major databases. Thank you for checking against Oracle sir, I do not have that option at the moment. Jarek ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/#review5344 ----------------------------------------------------------- On 2012-02-28 14:43:52, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/ ----------------------------------------------------------- (Updated 2012-02-28 14:43:52) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- I've incorporated support for incremental import of free form queries to sqoop. This addresses bug SQOOP-444 . https://issues.apache.org/jira/browse/SQOOP-444 Diffs ----- /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1294667 /src/java/org/apache/sqoop/tool/ImportTool.java 1294667 Diff: https://reviews.apache.org/r/3961/diff Testing ------- ant -Dhadoopversion={20,23,100} test live testing on CDH3 and Microsoft SQL Server Thanks, Jarek
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3961/#review5404
        -----------------------------------------------------------

        Ship it!

        Looks good to me.

        • Bilung

        On 2012-02-28 14:43:52, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3961/

        -----------------------------------------------------------

        (Updated 2012-02-28 14:43:52)

        Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

        Summary

        -------

        I've incorporated support for incremental import of free form queries to sqoop.

        This addresses bug SQOOP-444.

        https://issues.apache.org/jira/browse/SQOOP-444

        Diffs

        -----

        /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1294667

        /src/java/org/apache/sqoop/tool/ImportTool.java 1294667

        Diff: https://reviews.apache.org/r/3961/diff

        Testing

        -------

        ant -Dhadoopversion={20,23,100} test

        live testing on CDH3 and Microsoft SQL Server

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/#review5404 ----------------------------------------------------------- Ship it! Looks good to me. Bilung On 2012-02-28 14:43:52, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/ ----------------------------------------------------------- (Updated 2012-02-28 14:43:52) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- I've incorporated support for incremental import of free form queries to sqoop. This addresses bug SQOOP-444 . https://issues.apache.org/jira/browse/SQOOP-444 Diffs ----- /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1294667 /src/java/org/apache/sqoop/tool/ImportTool.java 1294667 Diff: https://reviews.apache.org/r/3961/diff Testing ------- ant -Dhadoopversion={20,23,100} test live testing on CDH3 and Microsoft SQL Server Thanks, Jarek
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2012-02-28 19:39:11, Bilung Lee wrote:

        > Looks good to me.

        Updated patch is already attached to JIRA, please feel free to commit.

        • Jarek

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3961/#review5404
        -----------------------------------------------------------

        On 2012-02-28 14:43:52, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3961/

        -----------------------------------------------------------

        (Updated 2012-02-28 14:43:52)

        Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

        Summary

        -------

        I've incorporated support for incremental import of free form queries to sqoop.

        This addresses bug SQOOP-444.

        https://issues.apache.org/jira/browse/SQOOP-444

        Diffs

        -----

        /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1294667

        /src/java/org/apache/sqoop/tool/ImportTool.java 1294667

        Diff: https://reviews.apache.org/r/3961/diff

        Testing

        -------

        ant -Dhadoopversion={20,23,100} test

        live testing on CDH3 and Microsoft SQL Server

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - On 2012-02-28 19:39:11, Bilung Lee wrote: > Looks good to me. Updated patch is already attached to JIRA, please feel free to commit. Jarek ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/#review5404 ----------------------------------------------------------- On 2012-02-28 14:43:52, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3961/ ----------------------------------------------------------- (Updated 2012-02-28 14:43:52) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- I've incorporated support for incremental import of free form queries to sqoop. This addresses bug SQOOP-444 . https://issues.apache.org/jira/browse/SQOOP-444 Diffs ----- /src/test/com/cloudera/sqoop/TestIncrementalImport.java 1294667 /src/java/org/apache/sqoop/tool/ImportTool.java 1294667 Diff: https://reviews.apache.org/r/3961/diff Testing ------- ant -Dhadoopversion={20,23,100} test live testing on CDH3 and Microsoft SQL Server Thanks, Jarek
        Hide
        Bilung Lee added a comment -

        Patch committed. Thanks, Jarcec!

        Show
        Bilung Lee added a comment - Patch committed. Thanks, Jarcec!
        Bilung Lee made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #89 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/89/)
        SQOOP-444 Support incremental import for free form queries (Revision 1294802)

        Result = FAILURE
        blee : http://svn.apache.org/viewvc/?view=rev&rev=1294802
        Files :

        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/ImportTool.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestIncrementalImport.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #89 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/89/ ) SQOOP-444 Support incremental import for free form queries (Revision 1294802) Result = FAILURE blee : http://svn.apache.org/viewvc/?view=rev&rev=1294802 Files : /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/ImportTool.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestIncrementalImport.java
        Bilung Lee made changes -
        Fix Version/s 1.4.2-incubating [ 12320141 ]

          People

          • Assignee:
            Jarek Jarcec Cecho
            Reporter:
            Jarek Jarcec Cecho
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development