Sqoop
  1. Sqoop
  2. SQOOP-384 Sqoop is incompatible with Hadoop prior to 0.21
  3. SQOOP-412

Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.3.0, 1.4.0-incubating
    • Fix Version/s: 1.4.1-incubating
    • Component/s: codegen
    • Labels:

      Description

      We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).

        Activity

        Jarek Jarcec Cecho created issue -
        Jarek Jarcec Cecho made changes -
        Field Original Value New Value
        Fix Version/s 1.4.1-incubating [ 12318902 ]
        Jarek Jarcec Cecho made changes -
        Component/s codegen [ 12315202 ]
        Eric Wadsworth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 1.4.0-incubating [ 12317345 ]
        Affects Version/s 1.3.0 [ 12317344 ]
        Labels patch
        Hide
        Eric Wadsworth added a comment -

        This is the patch.

        Show
        Eric Wadsworth added a comment - This is the patch.
        Eric Wadsworth made changes -
        Eric Wadsworth made changes -
        Comment [ This provides the getInstances method for use by the apache sqoop code. ]
        Arvind Prabhakar made changes -
        Assignee Eric Wadsworth [ wad ]
        Hide
        Jarek Jarcec Cecho added a comment -

        Patch committed,
        thank you very much Eric!

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Patch committed, thank you very much Eric! Jarcec
        Jarek Jarcec Cecho made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6 #70 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/70/)
        SQOOP-412. Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances

        (Eric Wadsworth via Jarek Jarcec Cecho)

        jarcec : http://svn.apache.org/viewvc/?view=rev&rev=1221127
        Files :

        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/config/ConfigurationHelper.java
        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/metastore/JobStorageFactory.java
        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/SqoopTool.java
        Show
        Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #70 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/70/ ) SQOOP-412 . Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances (Eric Wadsworth via Jarek Jarcec Cecho) jarcec : http://svn.apache.org/viewvc/?view=rev&rev=1221127 Files : /incubator/sqoop/trunk/src/java/org/apache/sqoop/config/ConfigurationHelper.java /incubator/sqoop/trunk/src/java/org/apache/sqoop/metastore/JobStorageFactory.java /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/SqoopTool.java
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3272/
        -----------------------------------------------------------

        Review request for Sqoop and Arvind Prabhakar.

        Summary
        -------

        I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21.

        This addresses bug SQOOP-412.
        https://issues.apache.org/jira/browse/SQOOP-412

        Diffs


        /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303
        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303
        /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303
        /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303
        /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION
        /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION
        /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION
        /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303
        /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION
        /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303
        /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303
        /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303
        /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303
        /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303

        Diff: https://reviews.apache.org/r/3272/diff

        Testing
        -------

        I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23.

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3272/ ----------------------------------------------------------- Review request for Sqoop and Arvind Prabhakar. Summary ------- I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21. This addresses bug SQOOP-412 . https://issues.apache.org/jira/browse/SQOOP-412 Diffs /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303 /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303 /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303 /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303 /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303 Diff: https://reviews.apache.org/r/3272/diff Testing ------- I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23. Thanks, Jarek
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3272/#review4028
        -----------------------------------------------------------

        Ship it!

        +1

        Thanks for the patch Jarcec. The changes look good. I tested basic import against MySQL and Hadoop 0.20.205 and that seems to work.

        Please attach the patch to the JIRA and also update the "Bugs" field to SQOOP-413 as that is the relevant JIRA for this issue.

        • Arvind

        On 2011-12-20 17:09:54, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3272/

        -----------------------------------------------------------

        (Updated 2011-12-20 17:09:54)

        Review request for Sqoop and Arvind Prabhakar.

        Summary

        -------

        I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21.

        This addresses bug SQOOP-412.

        https://issues.apache.org/jira/browse/SQOOP-412

        Diffs

        -----

        /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303

        /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303

        /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303

        /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303

        /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION

        /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION

        /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION

        /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303

        /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION

        /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303

        /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303

        /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303

        /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303

        /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303

        Diff: https://reviews.apache.org/r/3272/diff

        Testing

        -------

        I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23.

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3272/#review4028 ----------------------------------------------------------- Ship it! +1 Thanks for the patch Jarcec. The changes look good. I tested basic import against MySQL and Hadoop 0.20.205 and that seems to work. Please attach the patch to the JIRA and also update the "Bugs" field to SQOOP-413 as that is the relevant JIRA for this issue. Arvind On 2011-12-20 17:09:54, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3272/ ----------------------------------------------------------- (Updated 2011-12-20 17:09:54) Review request for Sqoop and Arvind Prabhakar. Summary ------- I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21. This addresses bug SQOOP-412 . https://issues.apache.org/jira/browse/SQOOP-412 Diffs ----- /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303 /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303 /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303 /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303 /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303 /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303 /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303 Diff: https://reviews.apache.org/r/3272/diff Testing ------- I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23. Thanks, Jarek
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        11h 18m 1 Eric Wadsworth 19/Dec/11 19:23
        Patch Available Patch Available Resolved Resolved
        12h 5m 1 Jarek Jarcec Cecho 20/Dec/11 07:28

          People

          • Assignee:
            Eric Wadsworth
            Reporter:
            Jarek Jarcec Cecho
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development