Sqoop
  1. Sqoop
  2. SQOOP-796

Unable to use new Hadoop environment variables

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.3
    • Component/s: None
    • Labels:
      None

      Description

      Sqoop uses HADOOP_HOME to locate its dependencies, but if using YARN, then the use of HADOOP_HOME has been deprecated and may not sufficient to locate all of the dependencies. Sqoop needs to be able to use HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME when they are used instead of HADOOP_HOME.

      1. SQOOP-796.patch.1
        14 kB
        Sean Mackrory
      2. SQOOP-796.patch.2
        13 kB
        Sean Mackrory
      3. SQOOP-796.patch.3
        14 kB
        Sean Mackrory

        Issue Links

          Activity

          Hide
          Sean Mackrory added a comment - - edited

          This patch includes changes to the shell scripts which detect the environment variables or set defaults, the documentation (I've chosen to simply replace HADOOP_HOME with the 2 new variables since it's deprecated, even though it should still work), and the Java code. It allows Sqoop to use HADOOP_COMMON_HOME (for the hadoop executable) and HADOOP_MAPRED_HOME (for the hadoop-core library) if they are available, but it can still use HADOOP_HOME for both if necessary.

          edit: I still need to do a bit more testing on this - I had some trouble on YARN but it appears to be a problem with my cluster, not Sqoop. Will update when I'm sure...

          Show
          Sean Mackrory added a comment - - edited This patch includes changes to the shell scripts which detect the environment variables or set defaults, the documentation (I've chosen to simply replace HADOOP_HOME with the 2 new variables since it's deprecated, even though it should still work), and the Java code. It allows Sqoop to use HADOOP_COMMON_HOME (for the hadoop executable) and HADOOP_MAPRED_HOME (for the hadoop-core library) if they are available, but it can still use HADOOP_HOME for both if necessary. edit: I still need to do a bit more testing on this - I had some trouble on YARN but it appears to be a problem with my cluster, not Sqoop. Will update when I'm sure...
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Sean,
          thank you very much for your patch and changes. Would you mind uploading your patch to Apache Review board [1]?

          Jarcec

          Links:
          1: https://reviews.apache.org

          Show
          Jarek Jarcec Cecho added a comment - Hi Sean, thank you very much for your patch and changes. Would you mind uploading your patch to Apache Review board [1] ? Jarcec Links: 1: https://reviews.apache.org
          Hide
          Sean Mackrory added a comment -
          Show
          Sean Mackrory added a comment - Uploaded: https://reviews.apache.org/r/8685/
          Hide
          Sean Mackrory added a comment -

          Updated patch based on initial feedback from reviewboard.

          Show
          Sean Mackrory added a comment - Updated patch based on initial feedback from reviewboard.
          Hide
          Sean Mackrory added a comment -

          Implementing more feedback from Reviewboard (added deprecated command-line options at the end of the command-line list, and replaced additional references to Cloudera's Distribution of Hadoop with Apache Bigtop).

          Show
          Sean Mackrory added a comment - Implementing more feedback from Reviewboard (added deprecated command-line options at the end of the command-line list, and replaced additional references to Cloudera's Distribution of Hadoop with Apache Bigtop).
          Hide
          Jarek Jarcec Cecho added a comment -
          Show
          Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=79f941b7e14df9b555084a7eee05980221d05f9e Thank you Sean for your contribution! Jarcec
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #505 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/505/)
          SQOOP-796: Unable to use new Hadoop environment variables (Revision 79f941b7e14df9b555084a7eee05980221d05f9e)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=79f941b7e14df9b555084a7eee05980221d05f9e
          Files :

          • src/docs/man/common-args.txt
          • bin/sqoop
          • src/docs/user/common-args.txt
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/docs/user/tools.txt
          • src/java/org/apache/sqoop/SqoopOptions.java
          • bin/configure-sqoop
          • conf/sqoop-env-template.sh
          • src/java/org/apache/sqoop/orm/CompilationManager.java
          • src/docs/man/sqoop.txt
          • src/docs/user/help.txt
          • src/scripts/run-perftest.sh
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #505 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/505/ ) SQOOP-796 : Unable to use new Hadoop environment variables (Revision 79f941b7e14df9b555084a7eee05980221d05f9e) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=79f941b7e14df9b555084a7eee05980221d05f9e Files : src/docs/man/common-args.txt bin/sqoop src/docs/user/common-args.txt src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/docs/user/tools.txt src/java/org/apache/sqoop/SqoopOptions.java bin/configure-sqoop conf/sqoop-env-template.sh src/java/org/apache/sqoop/orm/CompilationManager.java src/docs/man/sqoop.txt src/docs/user/help.txt src/scripts/run-perftest.sh
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #702 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/702/)
          SQOOP-796: Unable to use new Hadoop environment variables (Revision 79f941b7e14df9b555084a7eee05980221d05f9e)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=79f941b7e14df9b555084a7eee05980221d05f9e
          Files :

          • src/java/org/apache/sqoop/orm/CompilationManager.java
          • src/docs/user/tools.txt
          • bin/sqoop
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/docs/man/sqoop.txt
          • src/scripts/run-perftest.sh
          • src/docs/man/common-args.txt
          • src/docs/user/common-args.txt
          • src/docs/user/help.txt
          • conf/sqoop-env-template.sh
          • bin/configure-sqoop
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #702 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/702/ ) SQOOP-796 : Unable to use new Hadoop environment variables (Revision 79f941b7e14df9b555084a7eee05980221d05f9e) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=79f941b7e14df9b555084a7eee05980221d05f9e Files : src/java/org/apache/sqoop/orm/CompilationManager.java src/docs/user/tools.txt bin/sqoop src/java/org/apache/sqoop/SqoopOptions.java src/docs/man/sqoop.txt src/scripts/run-perftest.sh src/docs/man/common-args.txt src/docs/user/common-args.txt src/docs/user/help.txt conf/sqoop-env-template.sh bin/configure-sqoop src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #493 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/493/)
          SQOOP-796: Unable to use new Hadoop environment variables (Revision 79f941b7e14df9b555084a7eee05980221d05f9e)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=79f941b7e14df9b555084a7eee05980221d05f9e
          Files :

          • bin/configure-sqoop
          • src/docs/man/common-args.txt
          • src/docs/man/sqoop.txt
          • bin/sqoop
          • src/docs/user/help.txt
          • conf/sqoop-env-template.sh
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/docs/user/common-args.txt
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/scripts/run-perftest.sh
          • src/java/org/apache/sqoop/orm/CompilationManager.java
          • src/docs/user/tools.txt
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #493 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/493/ ) SQOOP-796 : Unable to use new Hadoop environment variables (Revision 79f941b7e14df9b555084a7eee05980221d05f9e) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=79f941b7e14df9b555084a7eee05980221d05f9e Files : bin/configure-sqoop src/docs/man/common-args.txt src/docs/man/sqoop.txt bin/sqoop src/docs/user/help.txt conf/sqoop-env-template.sh src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/docs/user/common-args.txt src/java/org/apache/sqoop/SqoopOptions.java src/scripts/run-perftest.sh src/java/org/apache/sqoop/orm/CompilationManager.java src/docs/user/tools.txt
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #492 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/492/)
          SQOOP-796: Unable to use new Hadoop environment variables (Revision 79f941b7e14df9b555084a7eee05980221d05f9e)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=79f941b7e14df9b555084a7eee05980221d05f9e
          Files :

          • src/docs/user/tools.txt
          • bin/configure-sqoop
          • src/java/org/apache/sqoop/orm/CompilationManager.java
          • bin/sqoop
          • src/java/org/apache/sqoop/SqoopOptions.java
          • conf/sqoop-env-template.sh
          • src/docs/man/common-args.txt
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/docs/user/common-args.txt
          • src/docs/user/help.txt
          • src/docs/man/sqoop.txt
          • src/scripts/run-perftest.sh
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #492 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/492/ ) SQOOP-796 : Unable to use new Hadoop environment variables (Revision 79f941b7e14df9b555084a7eee05980221d05f9e) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=79f941b7e14df9b555084a7eee05980221d05f9e Files : src/docs/user/tools.txt bin/configure-sqoop src/java/org/apache/sqoop/orm/CompilationManager.java bin/sqoop src/java/org/apache/sqoop/SqoopOptions.java conf/sqoop-env-template.sh src/docs/man/common-args.txt src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/docs/user/common-args.txt src/docs/user/help.txt src/docs/man/sqoop.txt src/scripts/run-perftest.sh

            People

            • Assignee:
              Sean Mackrory
              Reporter:
              Sean Mackrory
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development