Sqoop
  1. Sqoop
  2. SQOOP-1260

HADOOP_MAPRED_HOME should be defaulted correctly

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      Hadoop 2.x

      Description

      When running Sqoop on Hadoop 2.x which has been installed using the BigTop layout (with a custom hadoop-layout.sh), we run into issues in a setup where only Hadoop and sqoop are installed with the failure in the compilation of the generated Java file.

      This is because in Sqoop, the HADOOP_MAPRED_HOME is defaulted to HADOOP_HOME (/usr/lib/hadoop by default) when the variable is not set.

      This results in the mapreduce libraries not being added to the classpath and hence the failure. If we have HBase installed, then we add the HBase classpath which causes the mapreduce libraries to be brought into the classpath and hence we don't see the issue.

      We should default HADOOP_MAPRED_HOME to /usr/lib/hadoop-mapreduce (the BigTop default) if the directory is present else default to HADOOP_HOME

      1. SQOOP-1260-2.patch
        0.7 kB
        Venkat Ranganathan
      2. SQOOP-1260.patch
        1 kB
        Venkat Ranganathan

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          56s 1 Venkat Ranganathan 13/Dec/13 21:09
          Patch Available Patch Available Resolved Resolved
          6d 1h 36m 1 Jarek Jarcec Cecho 19/Dec/13 22:45
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #833 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/833/)
          SQOOP-1260: HADOOP_MAPRED_HOME should be defaulted correctly (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=49c0bf9b2e654eb36a4dabd989c470788c7e9ed8)

          • bin/configure-sqoop
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #833 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/833/ ) SQOOP-1260 : HADOOP_MAPRED_HOME should be defaulted correctly (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=49c0bf9b2e654eb36a4dabd989c470788c7e9ed8 ) bin/configure-sqoop
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #868 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/868/)
          SQOOP-1260: HADOOP_MAPRED_HOME should be defaulted correctly (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=49c0bf9b2e654eb36a4dabd989c470788c7e9ed8)

          • bin/configure-sqoop
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #868 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/868/ ) SQOOP-1260 : HADOOP_MAPRED_HOME should be defaulted correctly (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=49c0bf9b2e654eb36a4dabd989c470788c7e9ed8 ) bin/configure-sqoop
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1070 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1070/)
          SQOOP-1260: HADOOP_MAPRED_HOME should be defaulted correctly (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=49c0bf9b2e654eb36a4dabd989c470788c7e9ed8)

          • bin/configure-sqoop
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1070 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1070/ ) SQOOP-1260 : HADOOP_MAPRED_HOME should be defaulted correctly (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=49c0bf9b2e654eb36a4dabd989c470788c7e9ed8 ) bin/configure-sqoop
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #874 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/874/)
          SQOOP-1260: HADOOP_MAPRED_HOME should be defaulted correctly (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=49c0bf9b2e654eb36a4dabd989c470788c7e9ed8)

          • bin/configure-sqoop
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #874 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/874/ ) SQOOP-1260 : HADOOP_MAPRED_HOME should be defaulted correctly (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=49c0bf9b2e654eb36a4dabd989c470788c7e9ed8 ) bin/configure-sqoop
          Jarek Jarcec Cecho made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 1.4.5 [ 12324765 ]
          Resolution Fixed [ 1 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you Venkat Ranganathan, your contribution is appreciated!

          Show
          Jarek Jarcec Cecho added a comment - Thank you Venkat Ranganathan , your contribution is appreciated!
          Hide
          ASF subversion and git services added a comment -

          Commit 49c0bf9b2e654eb36a4dabd989c470788c7e9ed8 in branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=49c0bf9 ]

          SQOOP-1260: HADOOP_MAPRED_HOME should be defaulted correctly

          (Venkat Ranganathan via Jarek Jarcec Cecho)

          Show
          ASF subversion and git services added a comment - Commit 49c0bf9b2e654eb36a4dabd989c470788c7e9ed8 in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=49c0bf9 ] SQOOP-1260 : HADOOP_MAPRED_HOME should be defaulted correctly (Venkat Ranganathan via Jarek Jarcec Cecho)
          Hide
          Jarek Jarcec Cecho added a comment -

          +1

          Show
          Jarek Jarcec Cecho added a comment - +1
          Hide
          Venkat Ranganathan added a comment -

          Added a new patch. I have tested this patch also. Thanks for the review Jarek Jarcec Cecho. Very much appreciated.

          Show
          Venkat Ranganathan added a comment - Added a new patch. I have tested this patch also. Thanks for the review Jarek Jarcec Cecho . Very much appreciated.
          Venkat Ranganathan made changes -
          Attachment SQOOP-1260-2.patch [ 12619399 ]
          Venkat Ranganathan made changes -
          Remote Link This issue links to "Review board link (Web Link)" [ 13521 ]
          Venkat Ranganathan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Venkat Ranganathan [ venkatnrangan ]
          Venkat Ranganathan made changes -
          Field Original Value New Value
          Attachment SQOOP-1260.patch [ 12618684 ]
          Venkat Ranganathan created issue -

            People

            • Assignee:
              Venkat Ranganathan
              Reporter:
              Venkat Ranganathan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development