Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.4
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None
    1. SQOOP-1852.2.patch
      59 kB
      Gwen Shapira
    2. SQOOP-1852.1.patch
      58 kB
      Gwen Shapira
    3. SQOOP-1852.0.patch
      56 kB
      Gwen Shapira

      Issue Links

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop2-hadoop200 #651 (See https://builds.apache.org/job/Sqoop2-hadoop200/651/)
        SQOOP-1852: Sqoop2: Kafka connector supporting TO direction (abe: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e2e5aa8bf531115f8ce4055184a3b8892ce6beb4)

        • common-test/src/main/java/org/apache/sqoop/common/test/kafka/KafkaLocal.java
        • connector/connector-kafka/src/test/java/org/apache/sqoop/connector/kafka/TestConfigValidator.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConstants.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/ToJobConfiguration.java
        • test/src/main/java/org/apache/sqoop/test/testcases/KafkaConnectorTestCase.java
        • common-test/src/main/java/org/apache/sqoop/common/test/kafka/ZooKeeperLocal.java
        • connector/connector-kafka/src/main/resources/kafka-connector-config.properties
        • connector/pom.xml
        • connector/connector-kafka/src/test/java/org/apache/sqoop/connector/kafka/TestKafkaLoader.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaLoader.java
        • pom.xml
        • server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/LinkConfiguration.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnectorErrors.java
        • common-test/src/main/java/org/apache/sqoop/common/test/kafka/KafkaConsumer.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/ToJobConfig.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java
        • connector/connector-kafka/src/main/resources/sqoopconnector.properties
        • connector/connector-kafka/pom.xml
        • test/pom.xml
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToDestroyer.java
        • server/pom.xml
        • test/src/test/java/org/apache/sqoop/integration/connector/kafka/FromRDBMSToKafkaTest.java
        • common-test/src/main/java/org/apache/sqoop/common/test/kafka/TestUtil.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnector.java
        • common-test/pom.xml
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/LinkConfig.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop2-hadoop200 #651 (See https://builds.apache.org/job/Sqoop2-hadoop200/651/ ) SQOOP-1852 : Sqoop2: Kafka connector supporting TO direction (abe: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e2e5aa8bf531115f8ce4055184a3b8892ce6beb4 ) common-test/src/main/java/org/apache/sqoop/common/test/kafka/KafkaLocal.java connector/connector-kafka/src/test/java/org/apache/sqoop/connector/kafka/TestConfigValidator.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConstants.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/ToJobConfiguration.java test/src/main/java/org/apache/sqoop/test/testcases/KafkaConnectorTestCase.java common-test/src/main/java/org/apache/sqoop/common/test/kafka/ZooKeeperLocal.java connector/connector-kafka/src/main/resources/kafka-connector-config.properties connector/pom.xml connector/connector-kafka/src/test/java/org/apache/sqoop/connector/kafka/TestKafkaLoader.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaLoader.java pom.xml server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/LinkConfiguration.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnectorErrors.java common-test/src/main/java/org/apache/sqoop/common/test/kafka/KafkaConsumer.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/ToJobConfig.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java connector/connector-kafka/src/main/resources/sqoopconnector.properties connector/connector-kafka/pom.xml test/pom.xml connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToDestroyer.java server/pom.xml test/src/test/java/org/apache/sqoop/integration/connector/kafka/FromRDBMSToKafkaTest.java common-test/src/main/java/org/apache/sqoop/common/test/kafka/TestUtil.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnector.java common-test/pom.xml connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/LinkConfig.java
        Hide
        gwenshap Gwen Shapira added a comment -

        For users who may interested in using this connector (until we have actual docs!): note that due to SQOOP-1869, this Sqoop2 supports RDBMS->Kafka but not HDFS->Kafka.

        Show
        gwenshap Gwen Shapira added a comment - For users who may interested in using this connector (until we have actual docs!): note that due to SQOOP-1869 , this Sqoop2 supports RDBMS->Kafka but not HDFS->Kafka.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Sqoop2-hadoop100 #720 (See https://builds.apache.org/job/Sqoop2-hadoop100/720/)
        SQOOP-1852: Sqoop2: Kafka connector supporting TO direction (abe: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e2e5aa8bf531115f8ce4055184a3b8892ce6beb4)

        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/LinkConfiguration.java
        • connector/connector-kafka/src/main/resources/sqoopconnector.properties
        • connector/connector-kafka/src/main/resources/kafka-connector-config.properties
        • test/pom.xml
        • common-test/src/main/java/org/apache/sqoop/common/test/kafka/TestUtil.java
        • test/src/test/java/org/apache/sqoop/integration/connector/kafka/FromRDBMSToKafkaTest.java
        • common-test/src/main/java/org/apache/sqoop/common/test/kafka/KafkaConsumer.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/ToJobConfiguration.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConstants.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaLoader.java
        • common-test/src/main/java/org/apache/sqoop/common/test/kafka/KafkaLocal.java
        • common-test/pom.xml
        • common-test/src/main/java/org/apache/sqoop/common/test/kafka/ZooKeeperLocal.java
        • connector/connector-kafka/pom.xml
        • connector/connector-kafka/src/test/java/org/apache/sqoop/connector/kafka/TestConfigValidator.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnectorErrors.java
        • test/src/main/java/org/apache/sqoop/test/testcases/KafkaConnectorTestCase.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnector.java
        • pom.xml
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/LinkConfig.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/ToJobConfig.java
        • connector/connector-kafka/src/test/java/org/apache/sqoop/connector/kafka/TestKafkaLoader.java
        • server/pom.xml
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java
        • connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToDestroyer.java
        • server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java
        • connector/pom.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Sqoop2-hadoop100 #720 (See https://builds.apache.org/job/Sqoop2-hadoop100/720/ ) SQOOP-1852 : Sqoop2: Kafka connector supporting TO direction (abe: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e2e5aa8bf531115f8ce4055184a3b8892ce6beb4 ) connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/LinkConfiguration.java connector/connector-kafka/src/main/resources/sqoopconnector.properties connector/connector-kafka/src/main/resources/kafka-connector-config.properties test/pom.xml common-test/src/main/java/org/apache/sqoop/common/test/kafka/TestUtil.java test/src/test/java/org/apache/sqoop/integration/connector/kafka/FromRDBMSToKafkaTest.java common-test/src/main/java/org/apache/sqoop/common/test/kafka/KafkaConsumer.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/ToJobConfiguration.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConstants.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaLoader.java common-test/src/main/java/org/apache/sqoop/common/test/kafka/KafkaLocal.java common-test/pom.xml common-test/src/main/java/org/apache/sqoop/common/test/kafka/ZooKeeperLocal.java connector/connector-kafka/pom.xml connector/connector-kafka/src/test/java/org/apache/sqoop/connector/kafka/TestConfigValidator.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnectorErrors.java test/src/main/java/org/apache/sqoop/test/testcases/KafkaConnectorTestCase.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaConnector.java pom.xml connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/LinkConfig.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/configuration/ToJobConfig.java connector/connector-kafka/src/test/java/org/apache/sqoop/connector/kafka/TestKafkaLoader.java server/pom.xml connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToDestroyer.java server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java connector/pom.xml
        Hide
        abec Abraham Elmahrek added a comment -

        +1. Thanks Gwen Shapira!

        Show
        abec Abraham Elmahrek added a comment - +1. Thanks Gwen Shapira !
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit e2e5aa8bf531115f8ce4055184a3b8892ce6beb4 in sqoop's branch refs/heads/sqoop2 from Abraham Elmahrek
        [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=e2e5aa8 ]

        SQOOP-1852: Sqoop2: Kafka connector supporting TO direction

        (Gwen Shapira via Abraham Elmahrek)

        Show
        jira-bot ASF subversion and git services added a comment - Commit e2e5aa8bf531115f8ce4055184a3b8892ce6beb4 in sqoop's branch refs/heads/sqoop2 from Abraham Elmahrek [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=e2e5aa8 ] SQOOP-1852 : Sqoop2: Kafka connector supporting TO direction (Gwen Shapira via Abraham Elmahrek)
        Hide
        sqoop-qa Sqoop QA bot added a comment -

        Testing file SQOOP-1852.2.patch against branch sqoop2 took 0:24:26.702894.

        Overall: +1 all checks pass

        SUCCESS: Clean was successful
        SUCCESS: Patch applied correctly
        SUCCESS: Patch add/modify test case
        SUCCESS: Patch compiled
        SUCCESS: All unit tests passed
        SUCCESS: All integration tests passed

        Console output is available here.

        This message is automatically generated.

        Show
        sqoop-qa Sqoop QA bot added a comment - Testing file SQOOP-1852.2.patch against branch sqoop2 took 0:24:26.702894. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch add/modify test case SUCCESS: Patch compiled SUCCESS: All unit tests passed SUCCESS: All integration tests passed Console output is available here . This message is automatically generated.
        Hide
        gwenshap Gwen Shapira added a comment -

        Attached new version with fixes to RB comments. Including randomized IPs in integration tests.

        Show
        gwenshap Gwen Shapira added a comment - Attached new version with fixes to RB comments. Including randomized IPs in integration tests.
        Hide
        sqoop-qa Sqoop QA bot added a comment -

        Testing file SQOOP-1852.1.patch against branch sqoop2 took 0:24:19.137253.

        Overall: +1 all checks pass

        SUCCESS: Clean was successful
        SUCCESS: Patch applied correctly
        SUCCESS: Patch add/modify test case
        SUCCESS: Patch compiled
        SUCCESS: All unit tests passed
        SUCCESS: All integration tests passed

        Console output is available here.

        This message is automatically generated.

        Show
        sqoop-qa Sqoop QA bot added a comment - Testing file SQOOP-1852.1.patch against branch sqoop2 took 0:24:19.137253. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch add/modify test case SUCCESS: Patch compiled SUCCESS: All unit tests passed SUCCESS: All integration tests passed Console output is available here . This message is automatically generated.
        Hide
        gwenshap Gwen Shapira added a comment -
        • added integration test
        • fixed RB comments by Abe. Good call on CSV validator!
        Show
        gwenshap Gwen Shapira added a comment - added integration test fixed RB comments by Abe. Good call on CSV validator!
        Show
        gwenshap Gwen Shapira added a comment - https://reviews.apache.org/r/28802/
        Hide
        gwenshap Gwen Shapira added a comment -

        Note:
        Don't commit yet
        It passes unit tests, but I'd like to give it few real-world tries.

        Show
        gwenshap Gwen Shapira added a comment - Note: Don't commit yet It passes unit tests, but I'd like to give it few real-world tries.

          People

          • Assignee:
            gwenshap Gwen Shapira
            Reporter:
            gwenshap Gwen Shapira
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development