Sqoop
  1. Sqoop
  2. SQOOP-864

Sqoop2: Introduce ETL context objects

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.1
    • Fix Version/s: 1.99.2
    • Component/s: None
    • Labels:
      None

      Description

      I've came to a need to add new parameter into Extractor.extract and Loader.load methods. I've realized that doing so will break backward compatibility as both classes are part of public API for connectors. Right now it's not an issue as that API not stable yet, but it would be nice to create more generic approach that will allow us to extend it in the future without breaking backward compatibility.

      1. bugSQOOP-864.patch
        82 kB
        Jarek Jarcec Cecho
      2. bugSQOOP-864.patch
        82 kB
        Jarek Jarcec Cecho
      3. bugSQOOP-864.patch
        83 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Jarek Jarcec Cecho created issue -
          Jarek Jarcec Cecho made changes -
          Field Original Value New Value
          Link This issue blocks SQOOP-863 [ SQOOP-863 ]
          Jarek Jarcec Cecho made changes -
          Remote Link This issue links to "Review board (Web Link)" [ 12003 ]
          Jarek Jarcec Cecho made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jarek Jarcec Cecho made changes -
          Attachment bugSQOOP-864.patch [ 12568068 ]
          Jarek Jarcec Cecho made changes -
          Attachment bugSQOOP-864.patch [ 12568634 ]
          Jarek Jarcec Cecho made changes -
          Attachment bugSQOOP-864.patch [ 12568769 ]
          Hide
          Cheolsoo Park added a comment -

          +1. Committed to sqoop2.

          Show
          Cheolsoo Park added a comment - +1. Committed to sqoop2.
          Cheolsoo Park made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #174 (See https://builds.apache.org/job/Sqoop2-hadoop100/174/)
          SQOOP-864: Introduce ETL context objects (Revision ed9c514361b508b739c76df31d7719660f095bd3)

          Result = SUCCESS
          cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed9c514361b508b739c76df31d7719660f095bd3
          Files :

          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java
          • core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java
          • common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          • common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
          • spi/src/main/java/org/apache/sqoop/job/io/DataReader.java
          • common/src/main/java/org/apache/sqoop/etl/io/DataReader.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
          • spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
          • common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java
          • common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java
          • common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java
          • common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Loader.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
          • common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #174 (See https://builds.apache.org/job/Sqoop2-hadoop100/174/ ) SQOOP-864 : Introduce ETL context objects (Revision ed9c514361b508b739c76df31d7719660f095bd3) Result = SUCCESS cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed9c514361b508b739c76df31d7719660f095bd3 Files : connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java spi/src/main/java/org/apache/sqoop/job/io/DataReader.java common/src/main/java/org/apache/sqoop/etl/io/DataReader.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java spi/src/main/java/org/apache/sqoop/job/etl/Loader.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #187 (See https://builds.apache.org/job/Sqoop2-hadoop200/187/)
          SQOOP-864: Introduce ETL context objects (Revision ed9c514361b508b739c76df31d7719660f095bd3)

          Result = SUCCESS
          cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed9c514361b508b739c76df31d7719660f095bd3
          Files :

          • common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Loader.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java
          • common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java
          • spi/src/main/java/org/apache/sqoop/job/io/DataReader.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java
          • common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
          • common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java
          • common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java
          • common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java
          • common/src/main/java/org/apache/sqoop/etl/io/DataReader.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java
          • spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          • common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java
          • core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #187 (See https://builds.apache.org/job/Sqoop2-hadoop200/187/ ) SQOOP-864 : Introduce ETL context objects (Revision ed9c514361b508b739c76df31d7719660f095bd3) Result = SUCCESS cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ed9c514361b508b739c76df31d7719660f095bd3 Files : common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java spi/src/main/java/org/apache/sqoop/job/etl/Loader.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java spi/src/main/java/org/apache/sqoop/job/io/DataReader.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java common/src/main/java/org/apache/sqoop/job/etl/ActorContext.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java spi/src/main/java/org/apache/sqoop/job/etl/Extractor.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java spi/src/main/java/org/apache/sqoop/job/etl/Destroyer.java common/src/main/java/org/apache/sqoop/etl/io/DataReader.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java spi/src/main/java/org/apache/sqoop/job/io/DataWriter.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportInitializer.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java common/src/main/java/org/apache/sqoop/job/etl/InitializerContext.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java spi/src/main/java/org/apache/sqoop/job/etl/Partitioner.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development