Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2343

Shell Action should take Oozie Action config and setup HADOOP_CONF_DIR

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • trunk
    • 4.3.0
    • action
    • None

    Description

      Despite being not recommended, many users still run Hadoop ecosystem programs from the Shell Action (e.g. hive, beeline, sqoop, etc), even though we have Actions for those. One of the common problems we've been seeing here lately is that HADOOP_CONF_DIR is inherited from the NM, and is pointing to the NM's copy of the Hadoop *-site.xml files, which is often not well suited for client programs. This can lead to all kinds of weird problems, such as permissions errors, missing logs, etc.

      As a workaround, we've been telling users to set HADOOP_CONF_DIR to /etc/hadoop/conf, however, this has some disadvantages. Ideally, these client programs should pick up OOZIE_ACTION_CONF_XML instead. Given that OOZIE_ACTION_CONF_XML is an amalgamation of all of the Hadoop *-site.xml plus things Oozie and the user set, we can simply have the Shell Action copy-paste it to the correctly named *-site.xml files, and point HADOOP_CONF_DIR to there.

      Attachments

        1. OOZIE-2343.002.patch
          15 kB
          Robert Kanter
        2. OOZIE-2343.001.patch
          15 kB
          Robert Kanter

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rkanter Robert Kanter
            rkanter Robert Kanter
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment