Hadoop Common
  1. Hadoop Common
  2. HADOOP-9444

$var shell substitution in properties are not expanded in hadoop-policy.xml

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.1-alpha
    • Fix Version/s: 2.0.4-alpha
    • Component/s: conf
    • Labels:
      None
    • Environment:

      BigTop Kerberized cluster test environment

    • Hadoop Flags:
      Reviewed

      Description

      During BigTop 0.6.0 release test cycle, Roman Shaposhnik came around the following problem:

      013-03-26 15:37:03,573 FATAL
      org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
      NodeManager
      org.apache.hadoop.yarn.YarnException: Failed to Start
      org.apache.hadoop.yarn.server.nodemanager.NodeManager
              at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:199)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:322)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359)
      Caused by: org.apache.avro.AvroRuntimeException:
      java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:162)
              at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
              ... 3 more
      Caused by: java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
              at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:61)
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:199)
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:158)
              ... 4 more
      Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
      User yarn/ip-10-46-37-244.ec2.internal@BIGTOP (auth:KERBEROS) is not
      authorized for protocol interface
      org.apache.hadoop.yarn.server.api.ResourceTrackerPB, expected client
      Kerberos principal is yarn/ip-10-46-37-244.ec2.internal@BIGTOP
              at org.apache.hadoop.ipc.Client.call(Client.java:1235)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
              at $Proxy26.registerNodeManager(Unknown Source)
              at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
              ... 6 more
      
      

      The most significant part is
      User yarn/ip-10-46-37-244.ec2.internal@BIGTOP (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.yarn.server.api.ResourceTrackerPB indicating that ResourceTrackerPB hasn't been annotated with @KerberosInfo nor @TokenInfo

      1. YARN-509.patch.txt
        2 kB
        Roman Shaposhnik

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          Konstantin Boudnik, this doesn't seem to indicate missing securityInfo, but we can confirm. I need a little more information:

          • What is security.resourcetracker.protocol.acl set to in your hadoop-policy.xml?
          • What is yarn.nodemanager.principal in yarn-site.xml ?
          • Must already be the case, but can you verify if RMNMSecurityInfoClass.class and the text file org.apache.hadoop.security.SecurityInfo are on the classpath of ResourceManager?
          Show
          Vinod Kumar Vavilapalli added a comment - Konstantin Boudnik , this doesn't seem to indicate missing securityInfo, but we can confirm. I need a little more information: What is security.resourcetracker.protocol.acl set to in your hadoop-policy.xml? What is yarn.nodemanager.principal in yarn-site.xml ? Must already be the case, but can you verify if RMNMSecurityInfoClass.class and the text file org.apache.hadoop.security.SecurityInfo are on the classpath of ResourceManager?
          Hide
          Roman Shaposhnik added a comment -

          This is from Bigtop testing so I can make the cluster available for you (I'll need your public ssh key – please send it to me offline pref. PGP encoded). Now, to answer your questions:

          What is security.resourcetracker.protocol.acl set to in your hadoop-policy.xml?

          $

          {HADOOP_YARN_USER}

          which acording to the process environment translates to yarn

          What is yarn.nodemanager.principal in yarn-site.xml ?

          yarn/_HOST@BIGTOP

          RMNMSecurityInfoClass.class and the text file org.apache.hadoop.security.SecurityInfo are on the classpath of ResourceManager?

          Yes it is.

          Please let me know if you need any more info or if you'd like to get access to the cluster.

          Show
          Roman Shaposhnik added a comment - This is from Bigtop testing so I can make the cluster available for you (I'll need your public ssh key – please send it to me offline pref. PGP encoded). Now, to answer your questions: What is security.resourcetracker.protocol.acl set to in your hadoop-policy.xml? $ {HADOOP_YARN_USER} which acording to the process environment translates to yarn What is yarn.nodemanager.principal in yarn-site.xml ? yarn/_HOST@BIGTOP RMNMSecurityInfoClass.class and the text file org.apache.hadoop.security.SecurityInfo are on the classpath of ResourceManager? Yes it is. Please let me know if you need any more info or if you'd like to get access to the cluster.
          Hide
          Roman Shaposhnik added a comment -

          So it appears that $substitution in hadoop-policy.xml is broken.

          I propose that we simply change all the $value into '*' which will make the entries in hadoop-policy.xml fully consistent and also seems to fix the problem.

          Patch attached.

          Show
          Roman Shaposhnik added a comment - So it appears that $substitution in hadoop-policy.xml is broken. I propose that we simply change all the $value into '*' which will make the entries in hadoop-policy.xml fully consistent and also seems to fix the problem. Patch attached.
          Hide
          Konstantin Boudnik added a comment -

          I think it works around the problem rather than fixes it, doesn't it?

          Show
          Konstantin Boudnik added a comment - I think it works around the problem rather than fixes it, doesn't it?
          Hide
          Roman Shaposhnik added a comment -

          I totally agree that it needs to be investigated. That said, if we have to rush 2.0.4-alpha I'd say the proposed patch might be a reasonable workaround.

          Show
          Roman Shaposhnik added a comment - I totally agree that it needs to be investigated. That said, if we have to rush 2.0.4-alpha I'd say the proposed patch might be a reasonable workaround.
          Hide
          Roman Shaposhnik added a comment -

          Guys, I've updated the the description of the JIRA to be better reflect the latest findings. I'm leaving it as a blocker for now expecting somebody else to chime in and propose whether we apply a patch I provide or RELNOTE this if there's not enough time to get to the bottom of the issue.

          Show
          Roman Shaposhnik added a comment - Guys, I've updated the the description of the JIRA to be better reflect the latest findings. I'm leaving it as a blocker for now expecting somebody else to chime in and propose whether we apply a patch I provide or RELNOTE this if there's not enough time to get to the bottom of the issue.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575809/YARN-509.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/618//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/618//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575809/YARN-509.patch.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 eclipse:eclipse . The patch failed to build with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/618//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/618//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Moved to common.

          Apologies, this one slipped through.

          I see that HADOOP-9178 added this file in src/main/conf, but cannot trace the history back to when the variable thingie was added.

          +1 for the patch. eclipse:eclipse is failing for me even without the patch.

          Checking this in.

          Show
          Vinod Kumar Vavilapalli added a comment - Moved to common. Apologies, this one slipped through. I see that HADOOP-9178 added this file in src/main/conf, but cannot trace the history back to when the variable thingie was added. +1 for the patch. eclipse:eclipse is failing for me even without the patch. Checking this in.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I just committed this to trunk, branch-2 and branch-2.0.4-alpha. Thanks Roman!

          Show
          Vinod Kumar Vavilapalli added a comment - I just committed this to trunk, branch-2 and branch-2.0.4-alpha. Thanks Roman!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3538 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3538/)
          HADOOP-9444. Modify hadoop-policy.xml to replace unexpanded variables to a default value of '*'. Contributed by Roman Shaposhnik. (Revision 1462417)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1462417
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3538 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3538/ ) HADOOP-9444 . Modify hadoop-policy.xml to replace unexpanded variables to a default value of '*'. Contributed by Roman Shaposhnik. (Revision 1462417) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1462417 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575809/YARN-509.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2380//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2380//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575809/YARN-509.patch.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2380//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2380//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #169 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/169/)
          HADOOP-9444. Modify hadoop-policy.xml to replace unexpanded variables to a default value of '*'. Contributed by Roman Shaposhnik. (Revision 1462417)

          Result = FAILURE
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1462417
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #169 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/169/ ) HADOOP-9444 . Modify hadoop-policy.xml to replace unexpanded variables to a default value of '*'. Contributed by Roman Shaposhnik. (Revision 1462417) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1462417 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1358 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1358/)
          HADOOP-9444. Modify hadoop-policy.xml to replace unexpanded variables to a default value of '*'. Contributed by Roman Shaposhnik. (Revision 1462417)

          Result = FAILURE
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1462417
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1358 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1358/ ) HADOOP-9444 . Modify hadoop-policy.xml to replace unexpanded variables to a default value of '*'. Contributed by Roman Shaposhnik. (Revision 1462417) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1462417 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1386 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1386/)
          HADOOP-9444. Modify hadoop-policy.xml to replace unexpanded variables to a default value of '*'. Contributed by Roman Shaposhnik. (Revision 1462417)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1462417
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1386 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1386/ ) HADOOP-9444 . Modify hadoop-policy.xml to replace unexpanded variables to a default value of '*'. Contributed by Roman Shaposhnik. (Revision 1462417) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1462417 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
          Hide
          Roman Shaposhnik added a comment -

          Perfect! Thanks a million for your quick review and commit!

          Show
          Roman Shaposhnik added a comment - Perfect! Thanks a million for your quick review and commit!

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development