Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6803

AM registration could fail if event processing is delayed.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Steps to reproduce

      1. Submit application
      2. Delay application attempt AMLauch event processing
      3. Make AM register before AM Launch event is fired

      DefaultAMSProcessor#registerApplicationMaster client token

          if (UserGroupInformation.isSecurityEnabled()) {
            LOG.info("Setting client token master key");
            response.setClientToAMTokenMasterKey(java.nio.ByteBuffer.wrap(
                getRmContext().getClientToAMTokenSecretManager()
                .getMasterKey(applicationAttemptId).getEncoded()));
          }
      
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException: java.lang.NullPointerException
      	at org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.registerApplicationMaster(DefaultAMSProcessor.java:130)
      	at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.registerApplicationMaster(ApplicationMasterService.java:217)
      	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.registerApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:90)
      	at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:95)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
      
      	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)
      	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
      	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
      	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:978)
      	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
      	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1280)
      	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1733)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1729)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1660)
      

      Root Cause
      ClientToAMTokenSecretManagerInRM token master key is set only after AMLauch event is fired.

      AMLaunchedTransition

            // register the ClientTokenMasterKey after it is saved in the store,
            // otherwise client may hold an invalid ClientToken after RM restarts.
            if (UserGroupInformation.isSecurityEnabled()) {
              appAttempt.rmContext.getClientToAMTokenSecretManager()
                  .registerApplication(appAttempt.getAppAttemptId(),
                  appAttempt.getClientTokenMasterKey());
            }
      

        Attachments

        1. YARN-6803.001.patch
          2 kB
          Bibin A Chundatt
        2. YARN-6803.002.patch
          6 kB
          Bibin A Chundatt

          Issue Links

            Activity

              People

              • Assignee:
                bibinchundatt Bibin A Chundatt
                Reporter:
                bibinchundatt Bibin A Chundatt
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: