Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4264

Got ClassCastException when using mapreduce.history.server.delegationtoken.required=true

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.3
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      Oozie fails to run on branch-0.23 with the following exception. This only affects branch-0.23 not branch-2 or trunk.

      12/05/16 15:08:45 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/tgraves/.staging/job_1337177706246_0001java.lang.ClassCastException: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker cannot be cast to org.apache.hadoop.ipc.RpcInvocationHandler at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:330) at org.apache.hadoop.ipc.RPC.getServerAddress(RPC.java:320) at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getConnectAddress(MRClientProtocolPBClientImpl.java:108) at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:195) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:272) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:385) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)

      Note that this is caused because oozie passes in the options: -Dmapreduce.history.server.delegationtoken.required=true -Dmapreduce.history.server.delegationtoken.renewer="mr token"

        Activity

        Hide
        Thomas Graves added a comment -

        This appears to have been introduced by MAPREDUCE-4162 and HADOOP-8373 where a few of the Protobuf changes were pulled back from branch-2 to branch-0.23 to make the token changes easier. I believe it just missed one change needed in ProtoOverHadoopRpcEngine, which on the branch-2 is now common and renamed to be ProtobufRpcEngine. The Invoker class needs to implement RpcInvocationHandler instead of InvocationHandler.

        Show
        Thomas Graves added a comment - This appears to have been introduced by MAPREDUCE-4162 and HADOOP-8373 where a few of the Protobuf changes were pulled back from branch-2 to branch-0.23 to make the token changes easier. I believe it just missed one change needed in ProtoOverHadoopRpcEngine, which on the branch-2 is now common and renamed to be ProtobufRpcEngine. The Invoker class needs to implement RpcInvocationHandler instead of InvocationHandler.
        Hide
        Thomas Graves added a comment -

        initial patch that passes manual tests.

        Show
        Thomas Graves added a comment - initial patch that passes manual tests.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12527687/MAPREDUCE-4264-branch-0.23.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2390//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527687/MAPREDUCE-4264-branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2390//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        jenkins failed because it only applies to branch-0.23. Here is manual run:

        -1 overall.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 18 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        The javadoc warnings were not introduced by this jira - they are all pre-existing and have been fixed or excluded on trunk.

        Show
        Thomas Graves added a comment - jenkins failed because it only applies to branch-0.23. Here is manual run: -1 overall. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 18 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. The javadoc warnings were not introduced by this jira - they are all pre-existing and have been fixed or excluded on trunk.
        Hide
        Thomas Graves added a comment -

        I tested this manually by running wordcount with the -Dmapreduce.history.server.delegationtoken.required=true -Dmapreduce.history.server.delegationtoken.renewer="mr token" options. Without this fix you would get the cast exception above. I also verified that the IP put into the token file is correct - ie the one in the config file, in the case your host has IP aliasing turned on.

        Show
        Thomas Graves added a comment - I tested this manually by running wordcount with the -Dmapreduce.history.server.delegationtoken.required=true -Dmapreduce.history.server.delegationtoken.renewer="mr token" options. Without this fix you would get the cast exception above. I also verified that the IP put into the token file is correct - ie the one in the config file, in the case your host has IP aliasing turned on.
        Hide
        Robert Joseph Evans added a comment -

        The patch looks simple enough, and fixes the error. +1.

        Show
        Robert Joseph Evans added a comment - The patch looks simple enough, and fixes the error. +1.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #259 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/259/)
        MAPREDUCE-4264. ClassCastException with oozie. (tgraves via bobby) (Revision 1339478)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339478
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #259 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/259/ ) MAPREDUCE-4264 . ClassCastException with oozie. (tgraves via bobby) (Revision 1339478) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339478 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java

          People

          • Assignee:
            Thomas Graves
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development