Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1030

Enable column types Binary and Varbinary parsing in Sqoop for export

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.4
    • Fix Version/s: 1.4.4
    • Component/s: None
    • Labels:

      Description

      Enable parsing Binary and Varbinary column types which are represented as BytesWritable in Sqoop generated classes. Currently, Sqoop generated classes for database tables support serializing BytesWritable fields into files (while importing), but not the other way around for deserializing the records back when exporting from HDFS.
      We need to handle this for supporting Varbinary federation columns as well as Binary and Varbinary columns in MS Sql Server

      1. SQOOP-1030.1.patch
        1 kB
        Shuaishuai Nie
      2. SQOOP-1030.2.patch
        5 kB
        Shuaishuai Nie
      3. SQOOP-1030.3.patch
        6 kB
        Shuaishuai Nie

        Activity

        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Assigning to Shuaishuai Nie.

        Show
        jarcec Jarek Jarcec Cecho added a comment - Assigning to Shuaishuai Nie .
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Hi Shuaishuai Nie,
        thank you for working on this one! The change looks good to me, would you mind also providing test case for this new functionality?

        Jarcec

        Show
        jarcec Jarek Jarcec Cecho added a comment - Hi Shuaishuai Nie , thank you for working on this one! The change looks good to me, would you mind also providing test case for this new functionality? Jarcec
        Hide
        shuainie Shuaishuai Nie added a comment -

        Add test for binary and varbinary type in Microsoft SQL Server

        Show
        shuainie Shuaishuai Nie added a comment - Add test for binary and varbinary type in Microsoft SQL Server
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Hi Shuaishuai Nie,
        thank you for the quick turn around with new tests! Would you mind enhancing it a bit to validate that the inserted binary fragments is the expected value?

        Jarcec

        Show
        jarcec Jarek Jarcec Cecho added a comment - Hi Shuaishuai Nie , thank you for the quick turn around with new tests! Would you mind enhancing it a bit to validate that the inserted binary fragments is the expected value? Jarcec
        Hide
        shuainie Shuaishuai Nie added a comment -

        Thanks for the advice Jarek Jarcec Cecho. Added the row content check in the test

        Show
        shuainie Shuaishuai Nie added a comment - Thanks for the advice Jarek Jarcec Cecho . Added the row content check in the test
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        +1.

        Show
        jarcec Jarek Jarcec Cecho added a comment - +1.
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=b2d3a682adae553902897466555e2766ae665619

        Thank you Shuaaishuai for your contribution!

        Jarcec

        Show
        jarcec Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=b2d3a682adae553902897466555e2766ae665619 Thank you Shuaaishuai for your contribution! Jarcec
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop200 #637 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/637/)
        SQOOP-1030: Enable column types Binary and Varbinary parsing in Sqoop for export (Revision b2d3a682adae553902897466555e2766ae665619)

        Result = SUCCESS
        jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b2d3a682adae553902897466555e2766ae665619
        Files :

        • src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java
        • src/java/org/apache/sqoop/orm/ClassWriter.java
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #637 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/637/ ) SQOOP-1030 : Enable column types Binary and Varbinary parsing in Sqoop for export (Revision b2d3a682adae553902897466555e2766ae665619) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b2d3a682adae553902897466555e2766ae665619 Files : src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java src/java/org/apache/sqoop/orm/ClassWriter.java
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop23 #844 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/844/)
        SQOOP-1030: Enable column types Binary and Varbinary parsing in Sqoop for export (Revision b2d3a682adae553902897466555e2766ae665619)

        Result = SUCCESS
        jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b2d3a682adae553902897466555e2766ae665619
        Files :

        • src/java/org/apache/sqoop/orm/ClassWriter.java
        • src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #844 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/844/ ) SQOOP-1030 : Enable column types Binary and Varbinary parsing in Sqoop for export (Revision b2d3a682adae553902897466555e2766ae665619) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b2d3a682adae553902897466555e2766ae665619 Files : src/java/org/apache/sqoop/orm/ClassWriter.java src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop20 #636 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/636/)
        SQOOP-1030: Enable column types Binary and Varbinary parsing in Sqoop for export (Revision b2d3a682adae553902897466555e2766ae665619)

        Result = SUCCESS
        jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b2d3a682adae553902897466555e2766ae665619
        Files :

        • src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java
        • src/java/org/apache/sqoop/orm/ClassWriter.java
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #636 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/636/ ) SQOOP-1030 : Enable column types Binary and Varbinary parsing in Sqoop for export (Revision b2d3a682adae553902897466555e2766ae665619) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b2d3a682adae553902897466555e2766ae665619 Files : src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java src/java/org/apache/sqoop/orm/ClassWriter.java
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-ant-jdk-1.6-hadoop100 #640 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/640/)
        SQOOP-1030: Enable column types Binary and Varbinary parsing in Sqoop for export (Revision b2d3a682adae553902897466555e2766ae665619)

        Result = SUCCESS
        jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b2d3a682adae553902897466555e2766ae665619
        Files :

        • src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java
        • src/java/org/apache/sqoop/orm/ClassWriter.java
        Show
        hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #640 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/640/ ) SQOOP-1030 : Enable column types Binary and Varbinary parsing in Sqoop for export (Revision b2d3a682adae553902897466555e2766ae665619) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b2d3a682adae553902897466555e2766ae665619 Files : src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java src/java/org/apache/sqoop/orm/ClassWriter.java

          People

          • Assignee:
            shuainie Shuaishuai Nie
            Reporter:
            shuainie Shuaishuai Nie
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development