Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5246

Local task for map join submitted via oozie job fails on a secure HDFS

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      For a Hive query started by Oozie Hive action, the local task submitted for Mapjoin fails. The HDFS delegation token is not shared properly with the child JVM created for the local task.

      Oozie creates a delegation token for the Hive action and sets env variable HADOOP_TOKEN_FILE_LOCATION as well as mapreduce.job.credentials.binary config property. However this doesn't get passed down to the child JVM which causes the problem.
      This is similar issue addressed by HIVE-4343 which address the problem HiveServer2

      1. HIVE-5246.1.patch
        3 kB
        Prasad Mujumdar
      2. HIVE-5246-test.tar
        10 kB
        Prasad Mujumdar

        Activity

        Hide
        ashutoshc Ashutosh Chauhan added a comment -

        This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

        Show
        ashutoshc Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-h0.21 #2338 (See https://builds.apache.org/job/Hive-trunk-h0.21/2338/)
        HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2338 (See https://builds.apache.org/job/Hive-trunk-h0.21/2338/ ) HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2 #436 (See https://builds.apache.org/job/Hive-trunk-hadoop2/436/)
        HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #436 (See https://builds.apache.org/job/Hive-trunk-hadoop2/436/ ) HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Hide
        thejas Thejas M Nair added a comment -

        Patch committed to 0.12 branch.

        Show
        thejas Thejas M Nair added a comment - Patch committed to 0.12 branch.
        Hide
        thejas Thejas M Nair added a comment -

        This looks like a useful fix to include in 0.12 .

        Show
        thejas Thejas M Nair added a comment - This looks like a useful fix to include in 0.12 .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop1-ptest #169 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/169/)
        HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #169 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/169/ ) HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #102 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/102/)
        HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #102 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/102/ ) HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Hide
        brocknoland Brock Noland added a comment -

        Committed to trunk. Thank you for your contribution!

        Show
        brocknoland Brock Noland added a comment - Committed to trunk. Thank you for your contribution!
        Hide
        brocknoland Brock Noland added a comment -

        +1

        Show
        brocknoland Brock Noland added a comment - +1
        Hide
        hiveqa Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12603400/HIVE-5246.1.patch

        SUCCESS: +1 3097 tests passed

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/767/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/767/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        Show
        hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12603400/HIVE-5246.1.patch SUCCESS: +1 3097 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/767/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/767/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
        Hide
        prasadm Prasad Mujumdar added a comment -

        Reattached the patch file

        Show
        prasadm Prasad Mujumdar added a comment - Reattached the patch file
        Hide
        prasadm Prasad Mujumdar added a comment -
        Show
        prasadm Prasad Mujumdar added a comment - Review request on https://reviews.apache.org/r/14056/
        Hide
        prasadm Prasad Mujumdar added a comment -

        Test application to simulate oozie behavior

        Show
        prasadm Prasad Mujumdar added a comment - Test application to simulate oozie behavior

          People

          • Assignee:
            prasadm Prasad Mujumdar
            Reporter:
            prasadm Prasad Mujumdar
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development