Sqoop
  1. Sqoop
  2. SQOOP-1368

the configuration properties are reset in HBaseImportJob

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.6
    • Component/s: None
    • Labels:
      None

      Description

      if a configuration property is set from the command line. Then, the property will be reset in HBaseImportJob,

      For example,

      1. hbase.zookeeper.quorum is set in the command line
      2. when execute HBaseImportJob, hbase.zookeeper.quorum is reset as the value in hbase-site.xml. If hbase-site.xml is not in the classpath, hbase.zookeeper.quorum is reset to the value in hbase-default.xml.

      This issue prevents me from setting configuration properties in Oozie workflow. I can not use <configuration> element to set configurations in Sqoop Action.

      1. SQOOP-1368.patch
        0.6 kB
        Guodong Wang

        Activity

        Hide
        Guodong Wang added a comment -

        The issue is caused by this line.

            // Add HBase configuration files to this conf object.
            HBaseConfiguration.addHbaseResources(conf);
        

        I can provide a patch later.

        Show
        Guodong Wang added a comment - The issue is caused by this line. // Add HBase configuration files to this conf object. HBaseConfiguration.addHbaseResources(conf); I can provide a patch later.
        Hide
        Guodong Wang added a comment -

        upload patch.
        The patch is simple, just make sure no conf properties are reset during loading hbase configuration.

        use "ant clean test" to run the UT in trunk. No UT regression.

        Show
        Guodong Wang added a comment - upload patch. The patch is simple, just make sure no conf properties are reset during loading hbase configuration. use "ant clean test" to run the UT in trunk. No UT regression.
        Hide
        Jarek Jarcec Cecho added a comment -

        +1

        Show
        Jarek Jarcec Cecho added a comment - +1
        Hide
        ASF subversion and git services added a comment -

        Commit f05b4413a4ceae45fbc444aed7cd11ac6abb62ba in sqoop's branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=f05b441 ]

        SQOOP-1368: the configuration properties are reset in HBaseImportJob

        (Guodong Wang via Jarek Jarcec Cecho)

        Show
        ASF subversion and git services added a comment - Commit f05b4413a4ceae45fbc444aed7cd11ac6abb62ba in sqoop's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=f05b441 ] SQOOP-1368 : the configuration properties are reset in HBaseImportJob (Guodong Wang via Jarek Jarcec Cecho)
        Hide
        Jarek Jarcec Cecho added a comment -

        Thank you for your contribution Guodong Wang!

        Show
        Jarek Jarcec Cecho added a comment - Thank you for your contribution Guodong Wang !
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #906 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/906/)
        SQOOP-1368: the configuration properties are reset in HBaseImportJob (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f05b4413a4ceae45fbc444aed7cd11ac6abb62ba)

        • src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
        Show
        Hudson added a comment - FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #906 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/906/ ) SQOOP-1368 : the configuration properties are reset in HBaseImportJob (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f05b4413a4ceae45fbc444aed7cd11ac6abb62ba ) src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #912 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/912/)
        SQOOP-1368: the configuration properties are reset in HBaseImportJob (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f05b4413a4ceae45fbc444aed7cd11ac6abb62ba)

        • src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #912 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/912/ ) SQOOP-1368 : the configuration properties are reset in HBaseImportJob (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f05b4413a4ceae45fbc444aed7cd11ac6abb62ba ) src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1109 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1109/)
        SQOOP-1368: the configuration properties are reset in HBaseImportJob (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f05b4413a4ceae45fbc444aed7cd11ac6abb62ba)

        • src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1109 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1109/ ) SQOOP-1368 : the configuration properties are reset in HBaseImportJob (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f05b4413a4ceae45fbc444aed7cd11ac6abb62ba ) src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
        Hide
        Guodong Wang added a comment -

        I am very glad to contribute

        The jenkins job fails when building with hadoopversion20.
        I have checked the error, the hbase version is hbase;0.90.3-cdh3u1 in hadoopversion == 20. The version of hbase is too old in this conf.

        Jarek Jarcec Cecho, Shall we fix this build break?

        Show
        Guodong Wang added a comment - I am very glad to contribute The jenkins job fails when building with hadoopversion20. I have checked the error, the hbase version is hbase;0.90.3-cdh3u1 in hadoopversion == 20. The version of hbase is too old in this conf. Jarek Jarcec Cecho , Shall we fix this build break?
        Hide
        Jarek Jarcec Cecho added a comment -

        This is strange, I've tried running the build on my box with -Dhadoopversion=20 before committing this and it succeeded, yet now it's failing. I've must did something wrong. Yes let's create separate follow-up JIRA Guodong Wang and let's fix it there.

        Show
        Jarek Jarcec Cecho added a comment - This is strange, I've tried running the build on my box with -Dhadoopversion=20 before committing this and it succeeded, yet now it's failing. I've must did something wrong. Yes let's create separate follow-up JIRA Guodong Wang and let's fix it there.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #871 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/871/)
        SQOOP-1368: the configuration properties are reset in HBaseImportJob (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f05b4413a4ceae45fbc444aed7cd11ac6abb62ba)

        • src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #871 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/871/ ) SQOOP-1368 : the configuration properties are reset in HBaseImportJob (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f05b4413a4ceae45fbc444aed7cd11ac6abb62ba ) src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
        Hide
        Guodong Wang added a comment -

        I find these lines in the ant build script,

            <equals arg1="${hadoopversion}" arg2="20" />
            <then>
              <property name="hadoop.version" value="0.20.2-cdh3u1" />
              <property name="hbase94.version" value="0.90.3-cdh3u1" />
              <property name="zookeeper.version" value="3.3.3-cdh3u1" />
              <property name="hadoop.version.full" value="0.20" />
              <property name="hcatalog.version" value="0.13.0" />
              <property name="hbasecompatprofile" value="1" />
              <property name="avrohadoopprofile" value="1" />
            </then>
        

        The base version is set to 0.90.3-chd3u1 when hadoopversion=20, while merge function does not exist in 0.90.3.

        Show
        Guodong Wang added a comment - I find these lines in the ant build script, <equals arg1="${hadoopversion}" arg2="20" /> <then> <property name="hadoop.version" value="0.20.2-cdh3u1" /> <property name="hbase94.version" value="0.90.3-cdh3u1" /> <property name="zookeeper.version" value="3.3.3-cdh3u1" /> <property name="hadoop.version.full" value="0.20" /> <property name="hcatalog.version" value="0.13.0" /> <property name="hbasecompatprofile" value="1" /> <property name="avrohadoopprofile" value="1" /> </then> The base version is set to 0.90.3-chd3u1 when hadoopversion=20, while merge function does not exist in 0.90.3.
        Hide
        Jarek Jarcec Cecho added a comment -

        I think that we have two options:

        1) Use reflection APIs to call this method conditionally
        2) Bump up the HBase version

        Do you know in which version HBase added the required APIs Guodong Wang?

        Show
        Jarek Jarcec Cecho added a comment - I think that we have two options: 1) Use reflection APIs to call this method conditionally 2) Bump up the HBase version Do you know in which version HBase added the required APIs Guodong Wang ?
        Hide
        Guodong Wang added a comment -

        I prefer option 2.
        The merge function is added in base 0.92.
        Can we rebase hbase dependency to 0.92 in hadoop20 ?

        Show
        Guodong Wang added a comment - I prefer option 2. The merge function is added in base 0.92. Can we rebase hbase dependency to 0.92 in hadoop20 ?
        Hide
        Gwen Shapira added a comment -

        +1 for 0.92 in hadoop20.
        Even 0.94 is pretty old and everyone should have it by now.

        Show
        Gwen Shapira added a comment - +1 for 0.92 in hadoop20. Even 0.94 is pretty old and everyone should have it by now.
        Hide
        Jarek Jarcec Cecho added a comment -

        +1 to Gwen Shapira's sentiment.

        Show
        Jarek Jarcec Cecho added a comment - +1 to Gwen Shapira 's sentiment.

          People

          • Assignee:
            Guodong Wang
            Reporter:
            Guodong Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development