Hadoop Common
  1. Hadoop Common
  2. HADOOP-9444

$var shell substitution in properties are not expanded in hadoop-policy.xml

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.1-alpha
    • Fix Version/s: 2.0.4-alpha
    • Component/s: conf
    • Labels:
      None
    • Environment:

      BigTop Kerberized cluster test environment

    • Hadoop Flags:
      Reviewed

      Description

      During BigTop 0.6.0 release test cycle, Roman Shaposhnik came around the following problem:

      013-03-26 15:37:03,573 FATAL
      org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
      NodeManager
      org.apache.hadoop.yarn.YarnException: Failed to Start
      org.apache.hadoop.yarn.server.nodemanager.NodeManager
              at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:199)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:322)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359)
      Caused by: org.apache.avro.AvroRuntimeException:
      java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:162)
              at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
              ... 3 more
      Caused by: java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
              at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:61)
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:199)
              at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:158)
              ... 4 more
      Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
      User yarn/ip-10-46-37-244.ec2.internal@BIGTOP (auth:KERBEROS) is not
      authorized for protocol interface
      org.apache.hadoop.yarn.server.api.ResourceTrackerPB, expected client
      Kerberos principal is yarn/ip-10-46-37-244.ec2.internal@BIGTOP
              at org.apache.hadoop.ipc.Client.call(Client.java:1235)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
              at $Proxy26.registerNodeManager(Unknown Source)
              at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
              ... 6 more
      
      

      The most significant part is
      User yarn/ip-10-46-37-244.ec2.internal@BIGTOP (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.yarn.server.api.ResourceTrackerPB indicating that ResourceTrackerPB hasn't been annotated with @KerberosInfo nor @TokenInfo

      1. YARN-509.patch.txt
        2 kB
        Roman Shaposhnik

        Issue Links

          Activity

          Konstantin Boudnik created issue -
          Roman Shaposhnik made changes -
          Field Original Value New Value
          Attachment YARN-509.patch.txt [ 12575808 ]
          Roman Shaposhnik made changes -
          Attachment YARN-509.patch.txt [ 12575809 ]
          Roman Shaposhnik made changes -
          Attachment YARN-509.patch.txt [ 12575808 ]
          Roman Shaposhnik made changes -
          Summary ResourceTrackerPB misses KerberosInfo annotation which renders YARN unusable on secure clusters $var shell substitution in properties are not expanded in hadoop-policy.xml
          Roman Shaposhnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.0.4-alpha [ 12324137 ]
          Vinod Kumar Vavilapalli made changes -
          Project Hadoop YARN [ 12313722 ] Hadoop Common [ 12310240 ]
          Key YARN-509 HADOOP-9444
          Affects Version/s 2.0.1-alpha [ 12322467 ]
          Affects Version/s 2.0.1-alpha [ 12323270 ]
          Target Version/s 2.0.4-alpha [ 12324137 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 2.0.4-alpha [ 12324135 ]
          Fix Version/s 3.0.0 [ 12323268 ]
          Fix Version/s 2.0.4-alpha [ 12324137 ]
          Component/s conf [ 12310711 ]
          Component/s resourcemanager [ 12319322 ]
          Vinod Kumar Vavilapalli made changes -
          Assignee Roman Shaposhnik [ rvs ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Omkar Vinit Joshi made changes -
          Link This issue duplicates MAPREDUCE-3623 [ MAPREDUCE-3623 ]

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development