Sqoop
  1. Sqoop
  2. SQOOP-451

Add new options for format masks for date, time, and timestamp

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Add new options via which the user can specify format masks for date, time, and timestamp columns.

      The propose is to add pattern matching code to the toString() method of SqoopRecord so that when the output is written to files, it can be modified accordingly.

      1. SQOOP-451-2.patch
        35 kB
        Cheolsoo Park

        Issue Links

          Activity

          Cheolsoo Park made changes -
          Assignee Cheolsoo Park [ cheolsoo ]
          Hide
          Michael Bush added a comment -

          Would this apply to both import and export of "date/timestamp/timestamp with timezone" data types between Oracle and HDFS? That would be a great option!

          Show
          Michael Bush added a comment - Would this apply to both import and export of "date/timestamp/timestamp with timezone" data types between Oracle and HDFS? That would be a great option!
          Gavin made changes -
          Link This issue depends upon SQOOP-352 [ SQOOP-352 ]
          Gavin made changes -
          Link This issue depends on SQOOP-352 [ SQOOP-352 ]
          Cheolsoo Park made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Cheolsoo Park added a comment -

          I am cancelling patch for now until I get time to update it.

          Show
          Cheolsoo Park added a comment - I am cancelling patch for now until I get time to update it.
          Hide
          Jarek Jarcec Cecho added a comment -

          Please take your time, you have all the time in the world.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Please take your time, you have all the time in the world. Jarcec
          Hide
          Cheolsoo Park added a comment -

          Hi Jarcec, this was probably one of my first patches in old days. If I remember correctly, I tried to do quite a few clean-ups in test code in my patch. Since test code has since changed, I would expect many merge conflicts. Please allow some time for me to re-base this one.

          Show
          Cheolsoo Park added a comment - Hi Jarcec, this was probably one of my first patches in old days. If I remember correctly, I tried to do quite a few clean-ups in test code in my patch. Since test code has since changed, I would expect many merge conflicts. Please allow some time for me to re-base this one.
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Cheolsoo,
          please accept my deep apology for keeping this issue withnout notice for such long time. Would you mind rebasing your patch against current trunk and reuploading to review board?

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Cheolsoo, please accept my deep apology for keeping this issue withnout notice for such long time. Would you mind rebasing your patch against current trunk and reuploading to review board? Jarcec
          Cheolsoo Park made changes -
          Attachment SQOOP-451-2.patch [ 12530571 ]
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4327/
          -----------------------------------------------------------

          (Updated 2012-04-11 07:45:48.994371)

          Review request for Sqoop.

          Changes
          -------

          Improve the patch.

          Summary (updated)
          -------

          Add new options via which the user can specify format masks for date, time, and timestamp columns:

          --date-mask
          --time-mask
          --timestamp-mask

          To format text, I am using SimpleDateFormat.

          The changes include:

          1) Add the format mask options as Sqoop common options.
          2) Update ClassWriter so that SimpleDateFormat format() call can be generated in the toString() method.
          3) Update ClassWriter so that SimpleDateFormat parse() call can be generated in the __loadFromFields() method.
          4) Add new tests for format to ManagerCompatTest.
          5) Add new tests for parse to TestExport.

          In addition, I removed all try-catch blocks from TestExport that used to handle various date/time formats in Oracle db. Instead, output texts are formatted by SimpleDateFormat so that they are no longer needed.

          The patch is relatively big, but it is because many new tests are added. The actual changes for new options are not big.

          I would be very grateful if someone could review this patch.

          This addresses bug SQOOP-451.
          https://issues.apache.org/jira/browse/SQOOP-451

          Diffs (updated)


          /src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java 1311550
          /src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 1311550
          /src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1311550
          /src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1311550
          /src/test/com/cloudera/sqoop/manager/OracleExportTest.java 1311550
          /src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java 1311550
          /src/test/com/cloudera/sqoop/TestExport.java 1311550
          /src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1311550
          /src/java/org/apache/sqoop/orm/ClassWriter.java 1311550
          /src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 1311550
          /src/java/org/apache/sqoop/SqoopOptions.java 1311550

          Diff: https://reviews.apache.org/r/4327/diff

          Testing (updated)
          -------

          • Tests in ManagerCompatTest ensure that date/time/timestamp data are formatted correctly when being imported.
          • Tests in TestExport ensure that date/time/timestamp texts are parsed correctly when being exported.
          • Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4327/ ----------------------------------------------------------- (Updated 2012-04-11 07:45:48.994371) Review request for Sqoop. Changes ------- Improve the patch. Summary (updated) ------- Add new options via which the user can specify format masks for date, time, and timestamp columns: --date-mask --time-mask --timestamp-mask To format text, I am using SimpleDateFormat. The changes include: 1) Add the format mask options as Sqoop common options. 2) Update ClassWriter so that SimpleDateFormat format() call can be generated in the toString() method. 3) Update ClassWriter so that SimpleDateFormat parse() call can be generated in the __loadFromFields() method. 4) Add new tests for format to ManagerCompatTest. 5) Add new tests for parse to TestExport. In addition, I removed all try-catch blocks from TestExport that used to handle various date/time formats in Oracle db. Instead, output texts are formatted by SimpleDateFormat so that they are no longer needed. The patch is relatively big, but it is because many new tests are added. The actual changes for new options are not big. I would be very grateful if someone could review this patch. This addresses bug SQOOP-451 . https://issues.apache.org/jira/browse/SQOOP-451 Diffs (updated) /src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java 1311550 /src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 1311550 /src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1311550 /src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1311550 /src/test/com/cloudera/sqoop/manager/OracleExportTest.java 1311550 /src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java 1311550 /src/test/com/cloudera/sqoop/TestExport.java 1311550 /src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1311550 /src/java/org/apache/sqoop/orm/ClassWriter.java 1311550 /src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 1311550 /src/java/org/apache/sqoop/SqoopOptions.java 1311550 Diff: https://reviews.apache.org/r/4327/diff Testing (updated) ------- Tests in ManagerCompatTest ensure that date/time/timestamp data are formatted correctly when being imported. Tests in TestExport ensure that date/time/timestamp texts are parsed correctly when being exported. Ran ant test, ant test -Dthirdparty=true, and ant checkstyle. Thanks, Cheolsoo
          Cheolsoo Park made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4327/
          -----------------------------------------------------------

          Review request for Sqoop.

          Summary
          -------

          Add new options via which the user can specify format masks for date, time, and timestamp columns:

          --date-mask
          --time-mask
          --timestamp-mask

          To manipulate text from/to the DB, I am using SimpleDateFormat.

          The changes include:

          1) Add format mask options as Sqoop common options.
          2) Update ClassWriter so that SimpleDateFormat format() call can be generated in the toString() method.
          3) Update ClassWriter so that SimpleDateFormat parse() call can be generated in the __loadFromFields() method.
          4) Add new tests for import format to ManagerCompatTest and its subclasses.
          5) Add new tests for export parse to TestExport and its subclasses.
          6) Introduce regular expressions into OracleExportTest to get rid of try-catch blocks.
          (The format mask options do not format direct output from JDBC drivers.)
          7) Fix a minor bug in MySQLCompatTest regarding discarded fractional seconds.

          This addresses bug SQOOP-451.
          https://issues.apache.org/jira/browse/SQOOP-451

          Diffs


          ./src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 1301119
          ./src/java/org/apache/sqoop/SqoopOptions.java 1301119
          ./src/java/org/apache/sqoop/orm/ClassWriter.java 1301119
          ./src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1301119
          ./src/test/com/cloudera/sqoop/TestExport.java 1301119
          ./src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java 1301119
          ./src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 1301119
          ./src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1301119
          ./src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1301119
          ./src/test/com/cloudera/sqoop/manager/OracleExportTest.java 1301119
          ./src/test/com/cloudera/sqoop/testutil/ImportJobTestCase.java 1301119
          ./src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java 1301119

          Diff: https://reviews.apache.org/r/4327/diff

          Testing
          -------

          • Various format mask tests for import jobs are added to ManagerCompatTest.
          • Various format mask tests for export jobs are added to TestExport.
          • Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4327/ ----------------------------------------------------------- Review request for Sqoop. Summary ------- Add new options via which the user can specify format masks for date, time, and timestamp columns: --date-mask --time-mask --timestamp-mask To manipulate text from/to the DB, I am using SimpleDateFormat. The changes include: 1) Add format mask options as Sqoop common options. 2) Update ClassWriter so that SimpleDateFormat format() call can be generated in the toString() method. 3) Update ClassWriter so that SimpleDateFormat parse() call can be generated in the __loadFromFields() method. 4) Add new tests for import format to ManagerCompatTest and its subclasses. 5) Add new tests for export parse to TestExport and its subclasses. 6) Introduce regular expressions into OracleExportTest to get rid of try-catch blocks. (The format mask options do not format direct output from JDBC drivers.) 7) Fix a minor bug in MySQLCompatTest regarding discarded fractional seconds. This addresses bug SQOOP-451 . https://issues.apache.org/jira/browse/SQOOP-451 Diffs ./src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 1301119 ./src/java/org/apache/sqoop/SqoopOptions.java 1301119 ./src/java/org/apache/sqoop/orm/ClassWriter.java 1301119 ./src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1301119 ./src/test/com/cloudera/sqoop/TestExport.java 1301119 ./src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java 1301119 ./src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 1301119 ./src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1301119 ./src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1301119 ./src/test/com/cloudera/sqoop/manager/OracleExportTest.java 1301119 ./src/test/com/cloudera/sqoop/testutil/ImportJobTestCase.java 1301119 ./src/test/com/cloudera/sqoop/testutil/ManagerCompatTestCase.java 1301119 Diff: https://reviews.apache.org/r/4327/diff Testing ------- Various format mask tests for import jobs are added to ManagerCompatTest. Various format mask tests for export jobs are added to TestExport. Ran ant test, ant test -Dthirdparty=true, and ant checkstyle. Thanks, Cheolsoo
          Cheolsoo Park made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Cheolsoo Park made changes -
          Description Add new options via which the user can specify format masks for date, time, and timestamp columns.

          The propose is to add pattern matching code to the toString() method of SqoopRecorder so that when the output is written to files, it can be modified accordingly.
          Add new options via which the user can specify format masks for date, time, and timestamp columns.

          The propose is to add pattern matching code to the toString() method of SqoopRecord so that when the output is written to files, it can be modified accordingly.
          Cheolsoo Park made changes -
          Summary Add format string to select query for Oracle DB Add new options for format masks for date, time, and timestamp
          Description Oracle compatibility tests are fragile since the output format of timestamp from the DB varies depending on versions. To test different versions effectively, we should make the output more deterministic.

          This issue was originally raised by SQOOP-449, and using regular expressions in compatibility tests was suggested. But after discussion with Arvind and Bilung, we decided to add a format string for the date type to the select query for the Oracle DB so that the output format can be consistent.
          Add new options via which the user can specify format masks for date, time, and timestamp columns.

          The propose is to add pattern matching code to the toString() method of SqoopRecorder so that when the output is written to files, it can be modified accordingly.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4235/
          -----------------------------------------------------------

          (Updated 2012-03-08 01:52:58.860527)

          Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

          Changes
          -------

          Move fixupColumnTypes() from ClassWriter to OracleManager (i.e. no longer changing ClassWriter). It seems to fit better in OracleManager since that is Oracle-specific.

          Summary
          -------

          Oracle compatibility tests are fragile since the output format of timestamp from the DB varies depending on versions. To test different versions effectively, we should make the output more deterministic.

          This patch is not going to be submitted unless a new option is added to Sqoop so that new behaviors happen only if that option is enabled. I am posting my patch only to collect feedback.

          This addresses bug SQOOP-451.
          https://issues.apache.org/jira/browse/SQOOP-451

          Diffs (updated)


          ./src/java/com/cloudera/sqoop/mapreduce/db/DBConfiguration.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/manager/OracleManager.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1297783
          ./src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1297783
          ./src/test/com/cloudera/sqoop/manager/OracleManagerTest.java 1297783
          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1297783

          Diff: https://reviews.apache.org/r/4235/diff

          Testing
          -------

          ant test
          ant test -Dthirdparty=true

          Note that all the Oracle-specific methods in OracleCompatTest are removed since the same methods in ManagerCompatTest can be used as any other DBs.

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4235/ ----------------------------------------------------------- (Updated 2012-03-08 01:52:58.860527) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Changes ------- Move fixupColumnTypes() from ClassWriter to OracleManager (i.e. no longer changing ClassWriter). It seems to fit better in OracleManager since that is Oracle-specific. Summary ------- Oracle compatibility tests are fragile since the output format of timestamp from the DB varies depending on versions. To test different versions effectively, we should make the output more deterministic. This patch is not going to be submitted unless a new option is added to Sqoop so that new behaviors happen only if that option is enabled. I am posting my patch only to collect feedback. This addresses bug SQOOP-451 . https://issues.apache.org/jira/browse/SQOOP-451 Diffs (updated) ./src/java/com/cloudera/sqoop/mapreduce/db/DBConfiguration.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1297783 ./src/java/org/apache/sqoop/manager/OracleManager.java 1297783 ./src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1297783 ./src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1297783 ./src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1297783 ./src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1297783 ./src/test/com/cloudera/sqoop/manager/OracleManagerTest.java 1297783 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1297783 Diff: https://reviews.apache.org/r/4235/diff Testing ------- ant test ant test -Dthirdparty=true Note that all the Oracle-specific methods in OracleCompatTest are removed since the same methods in ManagerCompatTest can be used as any other DBs. Thanks, Cheolsoo
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4235/
          -----------------------------------------------------------

          Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

          Summary
          -------

          Oracle compatibility tests are fragile since the output format of timestamp from the DB varies depending on versions. To test different versions effectively, we should make the output more deterministic.

          This patch is not going to be submitted unless a new option is added to Sqoop so that new behaviors happen only if that option is enabled. I am posting my patch only to collect feedback.

          This addresses bug SQOOP-451.
          https://issues.apache.org/jira/browse/SQOOP-451

          Diffs


          ./src/java/com/cloudera/sqoop/mapreduce/db/DBConfiguration.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1297783
          ./src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/manager/OracleManager.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1297783
          ./src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1297783
          ./src/java/org/apache/sqoop/orm/ClassWriter.java 1297783
          ./src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1297783
          ./src/test/com/cloudera/sqoop/manager/OracleManagerTest.java 1297783
          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1297783

          Diff: https://reviews.apache.org/r/4235/diff

          Testing
          -------

          ant test
          ant test -Dthirdparty=true

          Note that all the Oracle-specific methods in OracleCompatTest are removed since the same methods in ManagerCompatTest can be used as any other DBs.

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4235/ ----------------------------------------------------------- Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- Oracle compatibility tests are fragile since the output format of timestamp from the DB varies depending on versions. To test different versions effectively, we should make the output more deterministic. This patch is not going to be submitted unless a new option is added to Sqoop so that new behaviors happen only if that option is enabled. I am posting my patch only to collect feedback. This addresses bug SQOOP-451 . https://issues.apache.org/jira/browse/SQOOP-451 Diffs ./src/java/com/cloudera/sqoop/mapreduce/db/DBConfiguration.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1297783 ./src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1297783 ./src/java/org/apache/sqoop/manager/OracleManager.java 1297783 ./src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1297783 ./src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1297783 ./src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1297783 ./src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1297783 ./src/java/org/apache/sqoop/orm/ClassWriter.java 1297783 ./src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1297783 ./src/test/com/cloudera/sqoop/manager/OracleManagerTest.java 1297783 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1297783 Diff: https://reviews.apache.org/r/4235/diff Testing ------- ant test ant test -Dthirdparty=true Note that all the Oracle-specific methods in OracleCompatTest are removed since the same methods in ManagerCompatTest can be used as any other DBs. Thanks, Cheolsoo
          Hide
          Cheolsoo Park added a comment -

          getColumnTypeNamesForTable() from SQOOP=352 is needed.

          Show
          Cheolsoo Park added a comment - getColumnTypeNamesForTable() from SQOOP=352 is needed.
          Cheolsoo Park made changes -
          Field Original Value New Value
          Link This issue depends on SQOOP-352 [ SQOOP-352 ]
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4172/
          -----------------------------------------------------------

          Review request for Sqoop and Bilung Lee.

          Summary
          -------

          Oracle compatibility tests are fragile since the output format of timestamp from the DB varies depending on versions. To test different versions effectively, we should make the output more deterministic.

          The changes include:
          1) Add format mask to the select query.
          2) Update OracleCompatTest accordingly.

          Note that date and timestamp are not distinguishable by the Oracle JDBC driver, so the same format is applied to date output as timestamp. Therefore, the testDate methods in OracleCompatTest expect timestamp output.

          This addresses bug SQOOP-451.
          https://issues.apache.org/jira/browse/SQOOP-451

          Diffs


          ./src/java/org/apache/sqoop/manager/OracleManager.java 1296761
          ./src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1296761

          Diff: https://reviews.apache.org/r/4172/diff

          Testing
          -------

          ant test
          ant test -Dthirdparty=true

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4172/ ----------------------------------------------------------- Review request for Sqoop and Bilung Lee. Summary ------- Oracle compatibility tests are fragile since the output format of timestamp from the DB varies depending on versions. To test different versions effectively, we should make the output more deterministic. The changes include: 1) Add format mask to the select query. 2) Update OracleCompatTest accordingly. Note that date and timestamp are not distinguishable by the Oracle JDBC driver, so the same format is applied to date output as timestamp. Therefore, the testDate methods in OracleCompatTest expect timestamp output. This addresses bug SQOOP-451 . https://issues.apache.org/jira/browse/SQOOP-451 Diffs ./src/java/org/apache/sqoop/manager/OracleManager.java 1296761 ./src/test/com/cloudera/sqoop/manager/OracleCompatTest.java 1296761 Diff: https://reviews.apache.org/r/4172/diff Testing ------- ant test ant test -Dthirdparty=true Thanks, Cheolsoo
          Cheolsoo Park created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Cheolsoo Park
            • Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development