Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5117

With security enabled HS delegation token renewer fails

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.4-alpha
    • Fix Version/s: 2.0.4-alpha
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It seems that the HSClientProtocolPBClientImpl should implement Closeable as per the attached stack trace. The problem can be observed on a cluster running the latest branch-2.0.4-alpha with MAPREDUCE-5088 applied on top. The easiest way to reproduce it is to run an oozie pig job:

      $ oozie job -oozie http://`hostname -f`:11000/oozie -run -DjobTracker=`hostname -f`:8032 -DnameNode=hdfs://`hostname -f`:17020 -DexamplesRoot=examples -config /tmp/examples/apps/pig/job.properties
      

      Please also note that I can successfully submit simple jobs (Pi/Sleep) from a command line using hadoop jar command. Thus it seems related to MAPREDUCE-5088 change.

      1. MAPREDUCE-5117.txt
        2 kB
        Siddharth Seth
      2. yarn.log
        25 kB
        Roman Shaposhnik

        Activity

        Roman Shaposhnik created issue -
        Roman Shaposhnik made changes -
        Field Original Value New Value
        Attachment yarn.log [ 12575975 ]
        Hide
        Daryn Sharp added a comment -

        I'm not sure it's related to MAPREDUCE-5088. The error is occurring in this block of code:

            MRClientProtocol histProxy = instantiateHistoryProxy(conf,
                SecurityUtil.getTokenServiceAddr(token));
            try {
              RenewDelegationTokenRequest request = Records
                  .newRecord(RenewDelegationTokenRequest.class);
              request.setDelegationToken(dToken);
              return histProxy.renewDelegationToken(request).getNextExpirationTime();
            } finally {
              stopHistoryProxy(histProxy);
            }
        

        It's blowing up in the finally clause that histProxy is not a proxy. The caught exception has been swallowed, but since histProxy isn't a proxy it's almost certainly failing on histProxy.renewDelegationToken.

        So the question is how is histProxy not a proxy??

        Show
        Daryn Sharp added a comment - I'm not sure it's related to MAPREDUCE-5088 . The error is occurring in this block of code: MRClientProtocol histProxy = instantiateHistoryProxy(conf, SecurityUtil.getTokenServiceAddr(token)); try { RenewDelegationTokenRequest request = Records .newRecord(RenewDelegationTokenRequest.class); request.setDelegationToken(dToken); return histProxy.renewDelegationToken(request).getNextExpirationTime(); } finally { stopHistoryProxy(histProxy); } It's blowing up in the finally clause that histProxy is not a proxy. The caught exception has been swallowed, but since histProxy isn't a proxy it's almost certainly failing on histProxy.renewDelegationToken. So the question is how is histProxy not a proxy??
        Hide
        Siddharth Seth added a comment -

        histProxy is actually a HSClientProtocolPBClientImpl - which contains the actual proxy. This, along with a couple other interfaces need to implement Closeable. Will submit a patch a little later.

        Show
        Siddharth Seth added a comment - histProxy is actually a HSClientProtocolPBClientImpl - which contains the actual proxy. This, along with a couple other interfaces need to implement Closeable. Will submit a patch a little later.
        Siddharth Seth made changes -
        Assignee Siddharth Seth [ sseth ]
        Hide
        Siddharth Seth added a comment -

        Changed MRClientProtocolPBClientImpl to implement Closeable. The alternate is to stop the history proxy using YarnRPC.

        Show
        Siddharth Seth added a comment - Changed MRClientProtocolPBClientImpl to implement Closeable. The alternate is to stop the history proxy using YarnRPC.
        Siddharth Seth made changes -
        Attachment MAPREDUCE-5117.txt [ 12576672 ]
        Siddharth Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Daryn Sharp added a comment -

        +1 Looks ok to me.

        Show
        Daryn Sharp added a comment - +1 Looks ok to me.
        Hide
        Konstantin Boudnik added a comment -

        I assume this should go to trunk and branch-2 as well.

        Show
        Konstantin Boudnik added a comment - I assume this should go to trunk and branch-2 as well.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12576672/MAPREDUCE-5117.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3491//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3491//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576672/MAPREDUCE-5117.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3491//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3491//console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Looks good, +1, checking it in.

        Show
        Vinod Kumar Vavilapalli added a comment - Looks good, +1, checking it in.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I just committed this to trunk, branch-2 and branch-2.0.4-alpha. Thanks Sid!

        Thanks to Roman and Daryn too for the help!

        Show
        Vinod Kumar Vavilapalli added a comment - I just committed this to trunk, branch-2 and branch-2.0.4-alpha. Thanks Sid! Thanks to Roman and Daryn too for the help!
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3553 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3553/)
        MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus fix failures in renewal of HistoryServer's delegations tokens. Contributed by Siddharth Seth. (Revision 1463828)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463828
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3553 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3553/ ) MAPREDUCE-5117 . Changed MRClientProtocolPBClientImpl to be closeable and thus fix failures in renewal of HistoryServer's delegations tokens. Contributed by Siddharth Seth. (Revision 1463828) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463828 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #174 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/174/)
        MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus fix failures in renewal of HistoryServer's delegations tokens. Contributed by Siddharth Seth. (Revision 1463828)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463828
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #174 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/174/ ) MAPREDUCE-5117 . Changed MRClientProtocolPBClientImpl to be closeable and thus fix failures in renewal of HistoryServer's delegations tokens. Contributed by Siddharth Seth. (Revision 1463828) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463828 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/)
        MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus fix failures in renewal of HistoryServer's delegations tokens. Contributed by Siddharth Seth. (Revision 1463828)

        Result = FAILURE
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463828
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/ ) MAPREDUCE-5117 . Changed MRClientProtocolPBClientImpl to be closeable and thus fix failures in renewal of HistoryServer's delegations tokens. Contributed by Siddharth Seth. (Revision 1463828) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463828 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1390 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/)
        MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus fix failures in renewal of HistoryServer's delegations tokens. Contributed by Siddharth Seth. (Revision 1463828)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463828
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1390 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/ ) MAPREDUCE-5117 . Changed MRClientProtocolPBClientImpl to be closeable and thus fix failures in renewal of HistoryServer's delegations tokens. Contributed by Siddharth Seth. (Revision 1463828) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463828 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development