Sqoop
  1. Sqoop
  2. SQOOP-384 Sqoop is incompatible with Hadoop prior to 0.21
  3. SQOOP-412

Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.3.0, 1.4.0-incubating
    • Fix Version/s: 1.4.1-incubating
    • Component/s: codegen
    • Labels:

      Description

      We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).

        Activity

        Hide
        Eric Wadsworth added a comment -

        This is the patch.

        Show
        Eric Wadsworth added a comment - This is the patch.
        Hide
        Jarek Jarcec Cecho added a comment -

        Patch committed,
        thank you very much Eric!

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Patch committed, thank you very much Eric! Jarcec
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #70 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/70/)
        SQOOP-412. Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances

        (Eric Wadsworth via Jarek Jarcec Cecho)

        jarcec : http://svn.apache.org/viewvc/?view=rev&rev=1221127
        Files :

        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/config/ConfigurationHelper.java
        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/metastore/JobStorageFactory.java
        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/SqoopTool.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #70 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/70/ ) SQOOP-412 . Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances (Eric Wadsworth via Jarek Jarcec Cecho) jarcec : http://svn.apache.org/viewvc/?view=rev&rev=1221127 Files : /incubator/sqoop/trunk/src/java/org/apache/sqoop/config/ConfigurationHelper.java /incubator/sqoop/trunk/src/java/org/apache/sqoop/metastore/JobStorageFactory.java /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/SqoopTool.java
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3272/
        -----------------------------------------------------------

        Review request for Sqoop and Arvind Prabhakar.

        Summary
        -------

        I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21.

        This addresses bug SQOOP-412.
        https://issues.apache.org/jira/browse/SQOOP-412

        Diffs


        /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303
        /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303
        /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303
        /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION
        /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION
        /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION
        /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303
        /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION
        /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303
        /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303
        /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303
        /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303

        Diff: https://reviews.apache.org/r/3272/diff

        Testing
        -------

        I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23.

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3272/ ----------------------------------------------------------- Review request for Sqoop and Arvind Prabhakar. Summary ------- I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21. This addresses bug SQOOP-412 . https://issues.apache.org/jira/browse/SQOOP-412 Diffs /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303 /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303 /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303 /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303 /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303 Diff: https://reviews.apache.org/r/3272/diff Testing ------- I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23. Thanks, Jarek
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3272/#review4028
        -----------------------------------------------------------

        Ship it!

        +1

        Thanks for the patch Jarcec. The changes look good. I tested basic import against MySQL and Hadoop 0.20.205 and that seems to work.

        Please attach the patch to the JIRA and also update the "Bugs" field to SQOOP-413 as that is the relevant JIRA for this issue.

        • Arvind

        On 2011-12-20 17:09:54, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3272/

        -----------------------------------------------------------

        (Updated 2011-12-20 17:09:54)

        Review request for Sqoop and Arvind Prabhakar.

        Summary

        -------

        I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21.

        This addresses bug SQOOP-412.

        https://issues.apache.org/jira/browse/SQOOP-412

        Diffs

        -----

        /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303

        /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303

        /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303

        /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION

        /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION

        /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION

        /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303

        /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION

        /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303

        /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303

        /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303

        /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303

        Diff: https://reviews.apache.org/r/3272/diff

        Testing

        -------

        I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23.

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3272/#review4028 ----------------------------------------------------------- Ship it! +1 Thanks for the patch Jarcec. The changes look good. I tested basic import against MySQL and Hadoop 0.20.205 and that seems to work. Please attach the patch to the JIRA and also update the "Bugs" field to SQOOP-413 as that is the relevant JIRA for this issue. Arvind On 2011-12-20 17:09:54, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3272/ ----------------------------------------------------------- (Updated 2011-12-20 17:09:54) Review request for Sqoop and Arvind Prabhakar. Summary ------- I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21. This addresses bug SQOOP-412 . https://issues.apache.org/jira/browse/SQOOP-412 Diffs ----- /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303 /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303 /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303 /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303 /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303 Diff: https://reviews.apache.org/r/3272/diff Testing ------- I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23. Thanks, Jarek

          People

          • Assignee:
            Eric Wadsworth
            Reporter:
            Jarek Jarcec Cecho
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development