Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2343

Shell Action should take Oozie Action config and setup HADOOP_CONF_DIR

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: trunk
    • Fix Version/s: 4.3.0
    • Component/s: action
    • Labels:
      None

      Description

      Despite being not recommended, many users still run Hadoop ecosystem programs from the Shell Action (e.g. hive, beeline, sqoop, etc), even though we have Actions for those. One of the common problems we've been seeing here lately is that HADOOP_CONF_DIR is inherited from the NM, and is pointing to the NM's copy of the Hadoop *-site.xml files, which is often not well suited for client programs. This can lead to all kinds of weird problems, such as permissions errors, missing logs, etc.

      As a workaround, we've been telling users to set HADOOP_CONF_DIR to /etc/hadoop/conf, however, this has some disadvantages. Ideally, these client programs should pick up OOZIE_ACTION_CONF_XML instead. Given that OOZIE_ACTION_CONF_XML is an amalgamation of all of the Hadoop *-site.xml plus things Oozie and the user set, we can simply have the Shell Action copy-paste it to the correctly named *-site.xml files, and point HADOOP_CONF_DIR to there.

      1. OOZIE-2343.002.patch
        15 kB
        Robert Kanter
      2. OOZIE-2343.001.patch
        15 kB
        Robert Kanter

        Issue Links

          Activity

          Hide
          rkanter Robert Kanter added a comment -

          Closing issue; Oozie 4.3.0 is released.

          Show
          rkanter Robert Kanter added a comment - Closing issue; Oozie 4.3.0 is released.
          Hide
          rkanter Robert Kanter added a comment -

          Thanks for the review Rohini. Committed to trunk!

          Show
          rkanter Robert Kanter added a comment - Thanks for the review Rohini. Committed to trunk!
          Hide
          rohini Rohini Palaniswamy added a comment -

          +1

          Show
          rohini Rohini Palaniswamy added a comment - +1
          Hide
          rkanter Robert Kanter added a comment -

          Ok, I think that's fair. The 002 patch switches the default to false.

          Show
          rkanter Robert Kanter added a comment - Ok, I think that's fair. The 002 patch switches the default to false .
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2343

          Cleaning local git workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          +1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . +1 the patch does adds/modifies 2 testcase(s)
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS - patch does not compile, cannot run testcases
          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/2517/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2343 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . +1 the patch does adds/modifies 2 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS - patch does not compile, cannot run testcases +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2517/
          Hide
          rohini Rohini Palaniswamy added a comment -

          Robert Kanter,
          I prefer that you make oozie.action.shell.setup.hadoop.conf.dir false by default. With Hadoop 2.x, tarballs are used to setup the client side *-site.xml files to support rolling upgrades. Hacking shell action to create the xml files does not make sense in those cases. For Pig/Hive we already added option (OOZIE-2205) to make the action xml not have the defaults and only ship overriden settings and let the Pig/Hive launcher jobs pick the local *-site.xml files from the mapreduce tarball. You can still have this option for users who do not have mapreduce client tarball in hdfs and instead use installation in node managers.

          Show
          rohini Rohini Palaniswamy added a comment - Robert Kanter , I prefer that you make oozie.action.shell.setup.hadoop.conf.dir false by default. With Hadoop 2.x, tarballs are used to setup the client side *-site.xml files to support rolling upgrades. Hacking shell action to create the xml files does not make sense in those cases. For Pig/Hive we already added option ( OOZIE-2205 ) to make the action xml not have the defaults and only ship overriden settings and let the Pig/Hive launcher jobs pick the local *-site.xml files from the mapreduce tarball. You can still have this option for users who do not have mapreduce client tarball in hdfs and instead use installation in node managers.

            People

            • Assignee:
              rkanter Robert Kanter
              Reporter:
              rkanter Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development