Sqoop
  1. Sqoop
  2. SQOOP-321

Support date/time columns for "--incremental append" option

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      Currently, when "--incremental append" option is used, only numeric columns are supported for check column.

      One workaround to "mimic" append mode with date/time values is
      1) select max(dt) from table into max_dt
      2) sqoop import --incremental lastmodified --check-column dt --last-value max_dt

      1. SQOOP-321.diff
        10 kB
        Bilung Lee

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1623/
        -----------------------------------------------------------

        Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar.

        Summary
        -------

        Currently, when "--incremental append" option is used, only numeric columns are supported for check column.

        This addresses bug SQOOP-321.
        https://issues.apache.org/jira/browse/SQOOP-321

        Diffs


        src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73
        src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc
        src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd
        src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309

        Diff: https://reviews.apache.org/r/1623/diff

        Testing
        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/ ----------------------------------------------------------- Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar. Summary ------- Currently, when "--incremental append" option is used, only numeric columns are supported for check column. This addresses bug SQOOP-321 . https://issues.apache.org/jira/browse/SQOOP-321 Diffs src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73 src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309 Diff: https://reviews.apache.org/r/1623/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1623/#review1610
        -----------------------------------------------------------

        Changes look good Bilung. I do have a backward compatibility concern noted below. Other than that, it will be good to give some examples of use of this feature in the user guide. The more specific examples you can give, the better it would be for the users.

        src/java/com/cloudera/sqoop/manager/ConnManager.java
        <https://reviews.apache.org/r/1623/#comment3639>

        This may be a compatibility issue for third party connectors. It is best to leave this method as is and introduce a new method that gets called when the check column type is not numeric.

        • Arvind

        On 2011-08-22 23:20:55, Bilung Lee wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1623/

        -----------------------------------------------------------

        (Updated 2011-08-22 23:20:55)

        Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar.

        Summary

        -------

        Currently, when "--incremental append" option is used, only numeric columns are supported for check column.

        This addresses bug SQOOP-321.

        https://issues.apache.org/jira/browse/SQOOP-321

        Diffs

        -----

        src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73

        src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc

        src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd

        src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309

        Diff: https://reviews.apache.org/r/1623/diff

        Testing

        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/#review1610 ----------------------------------------------------------- Changes look good Bilung. I do have a backward compatibility concern noted below. Other than that, it will be good to give some examples of use of this feature in the user guide. The more specific examples you can give, the better it would be for the users. src/java/com/cloudera/sqoop/manager/ConnManager.java < https://reviews.apache.org/r/1623/#comment3639 > This may be a compatibility issue for third party connectors. It is best to leave this method as is and introduce a new method that gets called when the check column type is not numeric. Arvind On 2011-08-22 23:20:55, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/ ----------------------------------------------------------- (Updated 2011-08-22 23:20:55) Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar. Summary ------- Currently, when "--incremental append" option is used, only numeric columns are supported for check column. This addresses bug SQOOP-321 . https://issues.apache.org/jira/browse/SQOOP-321 Diffs ----- src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73 src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309 Diff: https://reviews.apache.org/r/1623/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-08-24 03:29:53, Arvind Prabhakar wrote:

        > src/java/com/cloudera/sqoop/manager/ConnManager.java, lines 300-301

        > <https://reviews.apache.org/r/1623/diff/1/?file=34386#file34386line300>

        >

        > This may be a compatibility issue for third party connectors. It is best to leave this method as is and introduce a new method that gets called when the check column type is not numeric.

        Existing method is kept in new patch. Thanks!

        • Bilung

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1623/#review1610
        -----------------------------------------------------------

        On 2011-08-25 01:57:36, Bilung Lee wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1623/

        -----------------------------------------------------------

        (Updated 2011-08-25 01:57:36)

        Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar.

        Summary

        -------

        Currently, when "--incremental append" option is used, only numeric columns are supported for check column.

        This addresses bug SQOOP-321.

        https://issues.apache.org/jira/browse/SQOOP-321

        Diffs

        -----

        src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73

        src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc

        src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd

        src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309

        Diff: https://reviews.apache.org/r/1623/diff

        Testing

        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-08-24 03:29:53, Arvind Prabhakar wrote: > src/java/com/cloudera/sqoop/manager/ConnManager.java, lines 300-301 > < https://reviews.apache.org/r/1623/diff/1/?file=34386#file34386line300 > > > This may be a compatibility issue for third party connectors. It is best to leave this method as is and introduce a new method that gets called when the check column type is not numeric. Existing method is kept in new patch. Thanks! Bilung ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/#review1610 ----------------------------------------------------------- On 2011-08-25 01:57:36, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/ ----------------------------------------------------------- (Updated 2011-08-25 01:57:36) Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar. Summary ------- Currently, when "--incremental append" option is used, only numeric columns are supported for check column. This addresses bug SQOOP-321 . https://issues.apache.org/jira/browse/SQOOP-321 Diffs ----- src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73 src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309 Diff: https://reviews.apache.org/r/1623/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1623/
        -----------------------------------------------------------

        (Updated 2011-08-25 01:57:36.357578)

        Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar.

        Summary
        -------

        Currently, when "--incremental append" option is used, only numeric columns are supported for check column.

        This addresses bug SQOOP-321.
        https://issues.apache.org/jira/browse/SQOOP-321

        Diffs (updated)


        src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73
        src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc
        src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd
        src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309

        Diff: https://reviews.apache.org/r/1623/diff

        Testing
        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/ ----------------------------------------------------------- (Updated 2011-08-25 01:57:36.357578) Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar. Summary ------- Currently, when "--incremental append" option is used, only numeric columns are supported for check column. This addresses bug SQOOP-321 . https://issues.apache.org/jira/browse/SQOOP-321 Diffs (updated) src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73 src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309 Diff: https://reviews.apache.org/r/1623/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1623/#review1634
        -----------------------------------------------------------

        Ship it!

        +1

        • Arvind

        On 2011-08-25 01:57:36, Bilung Lee wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1623/

        -----------------------------------------------------------

        (Updated 2011-08-25 01:57:36)

        Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar.

        Summary

        -------

        Currently, when "--incremental append" option is used, only numeric columns are supported for check column.

        This addresses bug SQOOP-321.

        https://issues.apache.org/jira/browse/SQOOP-321

        Diffs

        -----

        src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73

        src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc

        src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd

        src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309

        Diff: https://reviews.apache.org/r/1623/diff

        Testing

        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/#review1634 ----------------------------------------------------------- Ship it! +1 Arvind On 2011-08-25 01:57:36, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/ ----------------------------------------------------------- (Updated 2011-08-25 01:57:36) Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar. Summary ------- Currently, when "--incremental append" option is used, only numeric columns are supported for check column. This addresses bug SQOOP-321 . https://issues.apache.org/jira/browse/SQOOP-321 Diffs ----- src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73 src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309 Diff: https://reviews.apache.org/r/1623/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1623/#review1633
        -----------------------------------------------------------

        Ship it!

        +1

        • Arvind

        On 2011-08-25 01:57:36, Bilung Lee wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1623/

        -----------------------------------------------------------

        (Updated 2011-08-25 01:57:36)

        Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar.

        Summary

        -------

        Currently, when "--incremental append" option is used, only numeric columns are supported for check column.

        This addresses bug SQOOP-321.

        https://issues.apache.org/jira/browse/SQOOP-321

        Diffs

        -----

        src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73

        src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc

        src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd

        src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309

        Diff: https://reviews.apache.org/r/1623/diff

        Testing

        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/#review1633 ----------------------------------------------------------- Ship it! +1 Arvind On 2011-08-25 01:57:36, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1623/ ----------------------------------------------------------- (Updated 2011-08-25 01:57:36) Review request for Sqoop, Ahmed Radwan and Arvind Prabhakar. Summary ------- Currently, when "--incremental append" option is used, only numeric columns are supported for check column. This addresses bug SQOOP-321 . https://issues.apache.org/jira/browse/SQOOP-321 Diffs ----- src/java/com/cloudera/sqoop/manager/ConnManager.java cea0a73 src/java/com/cloudera/sqoop/manager/OracleManager.java bad45dc src/java/com/cloudera/sqoop/tool/ImportTool.java 66e60bd src/test/com/cloudera/sqoop/TestIncrementalImport.java 96d4309 Diff: https://reviews.apache.org/r/1623/diff Testing ------- Thanks, Bilung
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Bilung!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Bilung!
        Hide
        Hudson added a comment -

        Integrated in Sqoop-jdk-1.6 #18 (See https://builds.apache.org/job/Sqoop-jdk-1.6/18/)
        SQOOP-321. Support date/time for incremental append imports.

        (Bilung Lee via Arvind Prabhakar)

        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1161404
        Files :

        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/ConnManager.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/OracleManager.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/tool/ImportTool.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestIncrementalImport.java
        Show
        Hudson added a comment - Integrated in Sqoop-jdk-1.6 #18 (See https://builds.apache.org/job/Sqoop-jdk-1.6/18/ ) SQOOP-321 . Support date/time for incremental append imports. (Bilung Lee via Arvind Prabhakar) arvind : http://svn.apache.org/viewvc/?view=rev&rev=1161404 Files : /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/ConnManager.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/OracleManager.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/tool/ImportTool.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestIncrementalImport.java

          People

          • Assignee:
            Bilung Lee
            Reporter:
            Bilung Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development