Hive
  1. Hive
  2. HIVE-5246

Local task for map join submitted via oozie job fails on a secure HDFS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      For a Hive query started by Oozie Hive action, the local task submitted for Mapjoin fails. The HDFS delegation token is not shared properly with the child JVM created for the local task.

      Oozie creates a delegation token for the Hive action and sets env variable HADOOP_TOKEN_FILE_LOCATION as well as mapreduce.job.credentials.binary config property. However this doesn't get passed down to the child JVM which causes the problem.
      This is similar issue addressed by HIVE-4343 which address the problem HiveServer2

      1. HIVE-5246.1.patch
        3 kB
        Prasad Mujumdar
      2. HIVE-5246-test.tar
        10 kB
        Prasad Mujumdar

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        10h 26m 1 Prasad Mujumdar 10/Sep/13 04:03
        Patch Available Patch Available Resolved Resolved
        7d 11h 42m 1 Brock Noland 17/Sep/13 15:45
        Resolved Resolved Closed Closed
        28d 8h 44m 1 Ashutosh Chauhan 16/Oct/13 00:29
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Ashutosh Chauhan added a comment -

        This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

        Show
        Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-h0.21 #2338 (See https://builds.apache.org/job/Hive-trunk-h0.21/2338/)
        HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2338 (See https://builds.apache.org/job/Hive-trunk-h0.21/2338/ ) HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2 #436 (See https://builds.apache.org/job/Hive-trunk-hadoop2/436/)
        HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #436 (See https://builds.apache.org/job/Hive-trunk-hadoop2/436/ ) HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Hide
        Thejas M Nair added a comment -

        Patch committed to 0.12 branch.

        Show
        Thejas M Nair added a comment - Patch committed to 0.12 branch.
        Thejas M Nair made changes -
        Fix Version/s 0.12.0 [ 12324312 ]
        Fix Version/s 0.13.0 [ 12324986 ]
        Hide
        Thejas M Nair added a comment -

        This looks like a useful fix to include in 0.12 .

        Show
        Thejas M Nair added a comment - This looks like a useful fix to include in 0.12 .
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop1-ptest #169 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/169/)
        HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #169 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/169/ ) HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #102 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/102/)
        HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #102 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/102/ ) HIVE-5246 - Local task for map join submitted via oozie job fails on a secure HDFS (Prasad Mujumdar via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524074 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
        Brock Noland made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.13.0 [ 12324986 ]
        Resolution Fixed [ 1 ]
        Hide
        Brock Noland added a comment -

        Committed to trunk. Thank you for your contribution!

        Show
        Brock Noland added a comment - Committed to trunk. Thank you for your contribution!
        Hide
        Brock Noland added a comment -

        +1

        Show
        Brock Noland added a comment - +1
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12603400/HIVE-5246.1.patch

        SUCCESS: +1 3097 tests passed

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/767/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/767/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12603400/HIVE-5246.1.patch SUCCESS: +1 3097 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/767/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/767/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
        Prasad Mujumdar made changes -
        Attachment HIVE-5246.1.patch [ 12603400 ]
        Hide
        Prasad Mujumdar added a comment -

        Reattached the patch file

        Show
        Prasad Mujumdar added a comment - Reattached the patch file
        Prasad Mujumdar made changes -
        Attachment HIVE-5246.1.patch [ 12602264 ]
        Prasad Mujumdar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Prasad Mujumdar added a comment -
        Show
        Prasad Mujumdar added a comment - Review request on https://reviews.apache.org/r/14056/
        Prasad Mujumdar made changes -
        Attachment HIVE-5246-test.tar [ 12602266 ]
        Hide
        Prasad Mujumdar added a comment -

        Test application to simulate oozie behavior

        Show
        Prasad Mujumdar added a comment - Test application to simulate oozie behavior
        Prasad Mujumdar made changes -
        Field Original Value New Value
        Attachment HIVE-5246.1.patch [ 12602264 ]
        Prasad Mujumdar created issue -

          People

          • Assignee:
            Prasad Mujumdar
            Reporter:
            Prasad Mujumdar
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development