Sqoop
  1. Sqoop
  2. SQOOP-313

Multiple column names to be included in --update-key argument with SQOOP export (update)

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      Currently the --update-key argument in SQOOP export (Updates) supports only one column name. But in most real time scenarios the target db table would have composite primary key hence in the SQL generated by SQOOP we may have to include more than one condition separated by AND in the WHERE clauses, ie the db UPDATE query generated should be like
      UPDATE .... SET ... WHERE PKColumn1 = '<value>' AND PKColumn2= '<value>' AND ...
      For this we need to modify --update-key argument to accomodate more that one column name which is separated using some separator like comma.

      1. SQOOP-313.patch
        32 kB
        Arvind Prabhakar
      2. SQOOP-313-1.patch
        33 kB
        Arvind Prabhakar

        Activity

        Hide
        Hudson added a comment -

        Integrated in Sqoop-jdk-1.6 #32 (See https://builds.apache.org/job/Sqoop-jdk-1.6/32/)
        SQOOP-313: Support for multiple column names for update keys

        (Arvind Prabhakar via Bilung Lee)

        blee : http://svn.apache.org/viewvc/?view=rev&rev=1177481
        Files :

        • /incubator/sqoop/trunk/src/docs/man/sqoop-export.txt
        • /incubator/sqoop/trunk/src/docs/user/export.txt
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/ConnManager.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/OracleManager.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestExportUpdate.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java
        Show
        Hudson added a comment - Integrated in Sqoop-jdk-1.6 #32 (See https://builds.apache.org/job/Sqoop-jdk-1.6/32/ ) SQOOP-313 : Support for multiple column names for update keys (Arvind Prabhakar via Bilung Lee) blee : http://svn.apache.org/viewvc/?view=rev&rev=1177481 Files : /incubator/sqoop/trunk/src/docs/man/sqoop-export.txt /incubator/sqoop/trunk/src/docs/user/export.txt /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/ConnManager.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/OracleManager.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestExportUpdate.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java
        Hide
        Bilung Lee added a comment -

        Committed. Thanks, Arvind!

        Show
        Bilung Lee added a comment - Committed. Thanks, Arvind!
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2115/#review2204
        -----------------------------------------------------------

        Ship it!

        Thanks for the update!

        • Bilung

        On 2011-09-30 01:49:27, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2115/

        -----------------------------------------------------------

        (Updated 2011-09-30 01:49:27)

        Review request for Sqoop and Bilung Lee.

        Summary

        -------

        Implemented the requested feature, added a few test cases and removed some existing checkstyle warnings from the code.

        This addresses bug SQOOP-313.

        https://issues.apache.org/jira/browse/SQOOP-313

        Diffs

        -----

        /src/docs/man/sqoop-export.txt 1177090

        /src/docs/user/export.txt 1177090

        /src/java/com/cloudera/sqoop/manager/ConnManager.java 1177090

        /src/java/com/cloudera/sqoop/manager/OracleManager.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java 1177090

        /src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java 1177090

        /src/test/com/cloudera/sqoop/TestExportUpdate.java 1177090

        /src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 1177090

        /src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 1177090

        Diff: https://reviews.apache.org/r/2115/diff

        Testing

        -------

        Ran unit and third-party tests. Also added a few new tests to exercise this functionality.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2115/#review2204 ----------------------------------------------------------- Ship it! Thanks for the update! Bilung On 2011-09-30 01:49:27, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2115/ ----------------------------------------------------------- (Updated 2011-09-30 01:49:27) Review request for Sqoop and Bilung Lee. Summary ------- Implemented the requested feature, added a few test cases and removed some existing checkstyle warnings from the code. This addresses bug SQOOP-313 . https://issues.apache.org/jira/browse/SQOOP-313 Diffs ----- /src/docs/man/sqoop-export.txt 1177090 /src/docs/user/export.txt 1177090 /src/java/com/cloudera/sqoop/manager/ConnManager.java 1177090 /src/java/com/cloudera/sqoop/manager/OracleManager.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java 1177090 /src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java 1177090 /src/test/com/cloudera/sqoop/TestExportUpdate.java 1177090 /src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 1177090 /src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 1177090 Diff: https://reviews.apache.org/r/2115/diff Testing ------- Ran unit and third-party tests. Also added a few new tests to exercise this functionality. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2115/
        -----------------------------------------------------------

        (Updated 2011-09-30 01:49:27.339886)

        Review request for Sqoop and Bilung Lee.

        Changes
        -------

        Thanks for the review Bilung. I have updated the patch based on your feedback. Also ran full unit and third-party tests to verify.

        Summary
        -------

        Implemented the requested feature, added a few test cases and removed some existing checkstyle warnings from the code.

        This addresses bug SQOOP-313.
        https://issues.apache.org/jira/browse/SQOOP-313

        Diffs (updated)


        /src/docs/man/sqoop-export.txt 1177090
        /src/docs/user/export.txt 1177090
        /src/java/com/cloudera/sqoop/manager/ConnManager.java 1177090
        /src/java/com/cloudera/sqoop/manager/OracleManager.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java 1177090
        /src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java 1177090
        /src/test/com/cloudera/sqoop/TestExportUpdate.java 1177090
        /src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 1177090
        /src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 1177090

        Diff: https://reviews.apache.org/r/2115/diff

        Testing
        -------

        Ran unit and third-party tests. Also added a few new tests to exercise this functionality.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2115/ ----------------------------------------------------------- (Updated 2011-09-30 01:49:27.339886) Review request for Sqoop and Bilung Lee. Changes ------- Thanks for the review Bilung. I have updated the patch based on your feedback. Also ran full unit and third-party tests to verify. Summary ------- Implemented the requested feature, added a few test cases and removed some existing checkstyle warnings from the code. This addresses bug SQOOP-313 . https://issues.apache.org/jira/browse/SQOOP-313 Diffs (updated) /src/docs/man/sqoop-export.txt 1177090 /src/docs/user/export.txt 1177090 /src/java/com/cloudera/sqoop/manager/ConnManager.java 1177090 /src/java/com/cloudera/sqoop/manager/OracleManager.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java 1177090 /src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java 1177090 /src/test/com/cloudera/sqoop/TestExportUpdate.java 1177090 /src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 1177090 /src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 1177090 Diff: https://reviews.apache.org/r/2115/diff Testing ------- Ran unit and third-party tests. Also added a few new tests to exercise this functionality. Thanks, Arvind
        Hide
        Arvind Prabhakar added a comment -

        Attached the updated patch.

        Show
        Arvind Prabhakar added a comment - Attached the updated patch.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2115/#review2200
        -----------------------------------------------------------

        Look good overall. One suggestion below (same but at different places).

        /src/java/com/cloudera/sqoop/manager/ConnManager.java
        <https://reviews.apache.org/r/2115/#comment5131>

        You may want to consider to make it an error if the length is 0.

        /src/java/com/cloudera/sqoop/manager/OracleManager.java
        <https://reviews.apache.org/r/2115/#comment5132>

        Same here.

        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java
        <https://reviews.apache.org/r/2115/#comment5133>

        Same here.

        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java
        <https://reviews.apache.org/r/2115/#comment5134>

        Same here.

        /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java
        <https://reviews.apache.org/r/2115/#comment5135>

        Same here.

        • Bilung

        On 2011-09-29 18:44:45, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2115/

        -----------------------------------------------------------

        (Updated 2011-09-29 18:44:45)

        Review request for Sqoop and Bilung Lee.

        Summary

        -------

        Implemented the requested feature, added a few test cases and removed some existing checkstyle warnings from the code.

        This addresses bug SQOOP-313.

        https://issues.apache.org/jira/browse/SQOOP-313

        Diffs

        -----

        /src/docs/man/sqoop-export.txt 1177090

        /src/docs/user/export.txt 1177090

        /src/java/com/cloudera/sqoop/manager/ConnManager.java 1177090

        /src/java/com/cloudera/sqoop/manager/OracleManager.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java 1177090

        /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java 1177090

        /src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java 1177090

        /src/test/com/cloudera/sqoop/TestExportUpdate.java 1177090

        /src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 1177090

        /src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 1177090

        Diff: https://reviews.apache.org/r/2115/diff

        Testing

        -------

        Ran unit and third-party tests. Also added a few new tests to exercise this functionality.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2115/#review2200 ----------------------------------------------------------- Look good overall. One suggestion below (same but at different places). /src/java/com/cloudera/sqoop/manager/ConnManager.java < https://reviews.apache.org/r/2115/#comment5131 > You may want to consider to make it an error if the length is 0. /src/java/com/cloudera/sqoop/manager/OracleManager.java < https://reviews.apache.org/r/2115/#comment5132 > Same here. /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java < https://reviews.apache.org/r/2115/#comment5133 > Same here. /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java < https://reviews.apache.org/r/2115/#comment5134 > Same here. /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java < https://reviews.apache.org/r/2115/#comment5135 > Same here. Bilung On 2011-09-29 18:44:45, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2115/ ----------------------------------------------------------- (Updated 2011-09-29 18:44:45) Review request for Sqoop and Bilung Lee. Summary ------- Implemented the requested feature, added a few test cases and removed some existing checkstyle warnings from the code. This addresses bug SQOOP-313 . https://issues.apache.org/jira/browse/SQOOP-313 Diffs ----- /src/docs/man/sqoop-export.txt 1177090 /src/docs/user/export.txt 1177090 /src/java/com/cloudera/sqoop/manager/ConnManager.java 1177090 /src/java/com/cloudera/sqoop/manager/OracleManager.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java 1177090 /src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java 1177090 /src/test/com/cloudera/sqoop/TestExportUpdate.java 1177090 /src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 1177090 /src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 1177090 Diff: https://reviews.apache.org/r/2115/diff Testing ------- Ran unit and third-party tests. Also added a few new tests to exercise this functionality. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2115/
        -----------------------------------------------------------

        Review request for Sqoop and Bilung Lee.

        Summary
        -------

        Implemented the requested feature, added a few test cases and removed some existing checkstyle warnings from the code.

        This addresses bug SQOOP-313.
        https://issues.apache.org/jira/browse/SQOOP-313

        Diffs


        /src/docs/man/sqoop-export.txt 1177090
        /src/docs/user/export.txt 1177090
        /src/java/com/cloudera/sqoop/manager/ConnManager.java 1177090
        /src/java/com/cloudera/sqoop/manager/OracleManager.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java 1177090
        /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java 1177090
        /src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java 1177090
        /src/test/com/cloudera/sqoop/TestExportUpdate.java 1177090
        /src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 1177090
        /src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 1177090

        Diff: https://reviews.apache.org/r/2115/diff

        Testing
        -------

        Ran unit and third-party tests. Also added a few new tests to exercise this functionality.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2115/ ----------------------------------------------------------- Review request for Sqoop and Bilung Lee. Summary ------- Implemented the requested feature, added a few test cases and removed some existing checkstyle warnings from the code. This addresses bug SQOOP-313 . https://issues.apache.org/jira/browse/SQOOP-313 Diffs /src/docs/man/sqoop-export.txt 1177090 /src/docs/user/export.txt 1177090 /src/java/com/cloudera/sqoop/manager/ConnManager.java 1177090 /src/java/com/cloudera/sqoop/manager/OracleManager.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/AvroExportMapper.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcUpdateExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/JdbcUpsertExportJob.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/OracleUpsertOutputFormat.java 1177090 /src/java/com/cloudera/sqoop/mapreduce/UpdateOutputFormat.java 1177090 /src/test/com/cloudera/sqoop/TestAvroImportExportRoundtrip.java 1177090 /src/test/com/cloudera/sqoop/TestExportUpdate.java 1177090 /src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 1177090 /src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 1177090 Diff: https://reviews.apache.org/r/2115/diff Testing ------- Ran unit and third-party tests. Also added a few new tests to exercise this functionality. Thanks, Arvind
        Hide
        Arvind Prabhakar added a comment -

        With this patch, Sqoop will support having multiple column names specified as a comma delimited list in the --update-key argument.

        Show
        Arvind Prabhakar added a comment - With this patch, Sqoop will support having multiple column names specified as a comma delimited list in the --update-key argument.

          People

          • Assignee:
            Arvind Prabhakar
            Reporter:
            Bejoy KS
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development