2019-06-18 10:34:01,616 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.tmpdir=/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir 2019-06-18 10:34:01,616 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.compiler= 2019-06-18 10:34:01,616 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.name=Linux 2019-06-18 10:34:01,616 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.arch=amd64 2019-06-18 10:34:01,616 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.version=3.10.0-862.11.6.el7.x86_64 2019-06-18 10:34:01,617 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.name=yarn 2019-06-18 10:34:01,617 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=/home/yarn 2019-06-18 10:34:01,617 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=/home/yarn 2019-06-18 10:34:01,618 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@582dcd35 2019-06-18 10:34:01,640 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:01,662 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:01,679 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00be, negotiated timeout = 40000 2019-06-18 10:34:01,696 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:01,725 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:01,728 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00be closed 2019-06-18 10:34:01,729 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560332747637_0003 of type yarn-service succeeded 2019-06-18 10:34:01,730 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00be 2019-06-18 10:34:01,731 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=hive OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560332747637_0003 submitted by user hive to unknown queue: llap APPID=application_1560332747637_0003 2019-06-18 10:34:01,732 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 1 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:01,735 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,735 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,736 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,736 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,737 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:01,738 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:01,738 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@4d0e1a9a 2019-06-18 10:34:01,742 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:01,742 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:01,745 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00bf, negotiated timeout = 40000 2019-06-18 10:34:01,746 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:01,759 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:01,760 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:01,762 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00bf closed 2019-06-18 10:34:01,762 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560396975962_0001 of type yarn-service succeeded 2019-06-18 10:34:01,763 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 1 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:01,766 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,767 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=hive OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560396975962_0001 submitted by user hive to unknown queue: llap APPID=application_1560396975962_0001 2019-06-18 10:34:01,770 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00bf 2019-06-18 10:34:01,772 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,774 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,774 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,775 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:01,775 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:01,776 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@5e37fb82 2019-06-18 10:34:01,779 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:01,780 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:01,782 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c0, negotiated timeout = 40000 2019-06-18 10:34:01,782 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:01,783 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:01,784 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:01,788 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c0 closed 2019-06-18 10:34:01,788 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560399221814_0001 of type yarn-service succeeded 2019-06-18 10:34:01,788 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c0 2019-06-18 10:34:01,790 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 2 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:01,791 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560399221814_0001 2019-06-18 10:34:01,796 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560399221814_0002 2019-06-18 10:34:01,796 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 3 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:01,806 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 4 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:01,808 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 5 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:01,810 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,811 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,812 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,816 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,817 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:01,817 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560399221814_0003 2019-06-18 10:34:01,817 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:01,829 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@750cd36d 2019-06-18 10:34:01,848 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:01,855 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:01,856 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560399221814_0004 2019-06-18 10:34:01,872 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c1, negotiated timeout = 40000 2019-06-18 10:34:01,878 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:01,901 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:01,902 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:01,908 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c1 closed 2019-06-18 10:34:01,908 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c1 2019-06-18 10:34:01,912 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560399221814_0005 of type yarn-service succeeded 2019-06-18 10:34:01,913 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 6 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:01,915 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,916 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,916 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,917 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,917 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:01,918 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:01,918 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@64c8fcfb 2019-06-18 10:34:01,937 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560399221814_0005 2019-06-18 10:34:01,937 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:01,938 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:01,944 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c2, negotiated timeout = 40000 2019-06-18 10:34:01,946 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:01,948 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:01,948 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:01,971 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c2 closed 2019-06-18 10:34:01,971 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560399221814_0006 of type yarn-service succeeded 2019-06-18 10:34:01,976 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Killed TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560399221814_0006 2019-06-18 10:34:01,981 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 7 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:01,981 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c2 2019-06-18 10:34:01,988 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,989 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,990 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:01,990 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:01,991 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:01,991 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:01,992 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@2bb717d7 2019-06-18 10:34:01,996 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:01,996 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:02,010 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c3, negotiated timeout = 40000 2019-06-18 10:34:02,011 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:02,013 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:02,014 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:02,015 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c3 closed 2019-06-18 10:34:02,016 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560399221814_0007 of type yarn-service succeeded 2019-06-18 10:34:02,016 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 1 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,017 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c3 2019-06-18 10:34:02,018 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560399221814_0007 2019-06-18 10:34:02,031 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 2 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,033 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,034 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,035 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,035 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,036 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:02,036 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:02,042 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@44ccd75c 2019-06-18 10:34:02,049 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560416851422_0001 2019-06-18 10:34:02,050 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:02,051 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:02,053 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c4, negotiated timeout = 40000 2019-06-18 10:34:02,053 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:02,071 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:02,071 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:02,075 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c4 closed 2019-06-18 10:34:02,075 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560416851422_0002 of type yarn-service succeeded 2019-06-18 10:34:02,076 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 3 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,078 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 4 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,080 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,081 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,081 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,083 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,084 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:02,084 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c4 2019-06-18 10:34:02,085 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:02,088 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@1ffd0114 2019-06-18 10:34:02,092 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560416851422_0002 2019-06-18 10:34:02,098 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:02,098 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:02,099 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=hive OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560416851422_0003 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560416851422_0003_000001 exited with exitCode: 12 Failing this attempt.Diagnostics: [2019-06-13 18:24:06.949]Exception from container-launch. Container id: container_1560416851422_0003_01_000001 Exit code: 12 [2019-06-13 18:24:07.081]Container exited with a non-zero exit code 12. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : $Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy17.finishApplicationMaster(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:606) at org.apache.spark.deploy.yarn.YarnRMClient.unregister(YarnRMClient.scala:89) at org.apache.spark.deploy.yarn.ApplicationMaster.unregister(ApplicationMaster.scala:384) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl$1.apply$mcV$sp(ApplicationMaster.scala:313) at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidApplicationMasterRequestException): Application doesn't exist in cache appattempt_1560416851422_0003_000001 at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.throwApplicationDoesNotExistInCacheException(ApplicationMasterService.java:353) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.finishApplicationMaster(ApplicationMasterService.java:319) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.finishApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:75) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:97) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at org.apache.hadoop.ipc.Client.call(Client.java:1443) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy16.finishApplicationMaster(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:92) ... 30 more 19/06/13 18:24:06 INFO ShutdownHookManager: Shutdown hook called [2019-06-13 18:24:07.082]Container exited with a non-zero exit code 12. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : $Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy17.finishApplicationMaster(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:606) at org.apache.spark.deploy.yarn.YarnRMClient.unregister(YarnRMClient.scala:89) at org.apache.spark.deploy.yarn.ApplicationMaster.unregister(ApplicationMaster.scala:384) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl$1.apply$mcV$sp(ApplicationMaster.scala:313) at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidApplicationMasterRequestException): Application doesn't exist in cache appattempt_1560416851422_0003_000001 at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.throwApplicationDoesNotExistInCacheException(ApplicationMasterService.java:353) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.finishApplicationMaster(ApplicationMasterService.java:319) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.finishApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:75) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:97) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at org.apache.hadoop.ipc.Client.call(Client.java:1443) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy16.finishApplicationMaster(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:92) ... 30 more 19/06/13 18:24:06 INFO ShutdownHookManager: Shutdown hook called For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560416851422_0003 Then click on links to logs of each attempt. . Failing the application. APPID=application_1560416851422_0003 2019-06-18 10:34:02,123 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c5, negotiated timeout = 40000 2019-06-18 10:34:02,125 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:02,144 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:02,144 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:02,149 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c5 closed 2019-06-18 10:34:02,149 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c5 2019-06-18 10:34:02,150 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560416851422_0004 of type yarn-service succeeded 2019-06-18 10:34:02,153 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 5 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,155 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,155 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,163 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,163 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,164 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:02,167 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Killed TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560416851422_0004 2019-06-18 10:34:02,169 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:02,172 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@5dfd31f4 2019-06-18 10:34:02,176 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:02,176 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:02,189 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c6, negotiated timeout = 40000 2019-06-18 10:34:02,189 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:02,191 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:02,192 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:02,194 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c6 closed 2019-06-18 10:34:02,194 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560416851422_0005 of type yarn-service succeeded 2019-06-18 10:34:02,195 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 6 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,197 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c6 2019-06-18 10:34:02,198 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560416851422_0005 2019-06-18 10:34:02,199 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,199 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,200 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,200 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,201 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:02,201 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:02,206 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@632241f5 2019-06-18 10:34:02,217 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:02,220 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:02,224 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c7, negotiated timeout = 40000 2019-06-18 10:34:02,224 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:02,236 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:02,238 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:02,240 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c7 closed 2019-06-18 10:34:02,240 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c7 2019-06-18 10:34:02,241 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560416851422_0006 of type yarn-service succeeded 2019-06-18 10:34:02,241 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=hive OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560416851422_0006 failed 5 times due to AM Container for appattempt_1560416851422_0006_000006 exited with exitCode: -1000 Failing this attempt.Diagnostics: [2019-06-13 18:29:01.318]Resource hdfs://ng02e8517-master-instance-4zau3mvg.novalocal:8020/hdp/apps/yarn/service-dep.tar.gz changed on src filesystem (expected 1560420820534, was 1560421507305 java.io.IOException: Resource hdfs://ng02e8517-master-instance-4zau3mvg.novalocal:8020/hdp/apps/yarn/service-dep.tar.gz changed on src filesystem (expected 1560420820534, was 1560421507305 at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:273) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:242) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:235) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:223) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560416851422_0006 Then click on links to logs of each attempt. . Failing the application. APPID=application_1560416851422_0006 2019-06-18 10:34:02,242 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 1 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,245 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,245 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,246 INFO client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:8050 2019-06-18 10:34:02,246 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:10200 2019-06-18 10:34:02,247 INFO zk.CuratorService (CuratorService.java:createCurator(268)) - Creating CuratorService with connection fixed ZK quorum "ng02e8517-master-instance-4zau3mvg.novalocal:2181" 2019-06-18 10:34:02,247 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(235)) - Starting 2019-06-18 10:34:02,248 INFO zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=ng02e8517-master-instance-4zau3mvg.novalocal:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@6df791a4 2019-06-18 10:34:02,256 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1032)) - Opening socket connection to server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-18 10:34:02,257 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(876)) - Socket connection established to ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, initiating session 2019-06-18 10:34:02,262 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1299)) - Session establishment complete on server ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30:2181, sessionid = 0x16b4b1397fd00c8, negotiated timeout = 40000 2019-06-18 10:34:02,264 INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED 2019-06-18 10:34:02,265 INFO client.ServiceClient (ServiceClient.java:cleanUpRegistryPath(673)) - Service 'llap0' doesn't exist at ZK registry path: /users/hive/services/yarn-service/llap0 2019-06-18 10:34:02,266 INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:backgroundOperationsLoop(821)) - backgroundOperationsLoop exiting 2019-06-18 10:34:02,267 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x16b4b1397fd00c8 closed 2019-06-18 10:34:02,267 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(519)) - EventThread shut down for session: 0x16b4b1397fd00c8 2019-06-18 10:34:02,268 INFO rmapp.RMAppImpl (RMAppImpl.java:appAdminClientCleanUp(1452)) - Type-specific cleanup of application application_1560421737895_0001 of type yarn-service succeeded 2019-06-18 10:34:02,269 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 2 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,270 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0001 2019-06-18 10:34:02,272 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 3 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,273 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0002 2019-06-18 10:34:02,275 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=hive OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0003 2019-06-18 10:34:02,278 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 20 for application: 4 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,278 INFO rmapp.RMAppImpl (RMAppImpl.java:recover(930)) - Recovering app: application_1560421737895_0004 with 1 attempts and final state = NONE 2019-06-18 10:34:02,279 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:recover(965)) - Recovering attempt: appattempt_1560421737895_0004_000001 with final state = NONE 2019-06-18 10:34:02,279 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1560421737895_0004_000001 2019-06-18 10:34:02,283 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1560421737895_0004_000001 2019-06-18 10:34:02,290 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1560421737895_0004 user: hive leaf-queue of parent: root #applications: 1 2019-06-18 10:34:02,291 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationOnRecovery(808)) - Accepted application application_1560421737895_0004 from user: hive, in queue: llap 2019-06-18 10:34:02,292 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0004_000001 State change from NEW to LAUNCHED on event = RECOVER 2019-06-18 10:34:02,292 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0004 State change from NEW to ACCEPTED on event = RECOVER 2019-06-18 10:34:02,293 INFO rmapp.RMAppImpl (RMAppImpl.java:recover(930)) - Recovering app: application_1560421737895_0005 with 1 attempts and final state = NONE 2019-06-18 10:34:02,294 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:recover(965)) - Recovering attempt: appattempt_1560421737895_0005_000001 with final state = NONE 2019-06-18 10:34:02,294 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1560421737895_0005_000001 2019-06-18 10:34:02,294 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1560421737895_0005_000001 2019-06-18 10:34:02,295 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1560421737895_0005 user: hive leaf-queue of parent: root #applications: 2 2019-06-18 10:34:02,295 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationOnRecovery(808)) - Accepted application application_1560421737895_0005 from user: hive, in queue: llap 2019-06-18 10:34:02,295 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0005_000001 State change from NEW to LAUNCHED on event = RECOVER 2019-06-18 10:34:02,295 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0005 State change from NEW to ACCEPTED on event = RECOVER 2019-06-18 10:34:02,298 INFO rmapp.RMAppImpl (RMAppImpl.java:recover(930)) - Recovering app: application_1560421737895_0006 with 1 attempts and final state = NONE 2019-06-18 10:34:02,298 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:recover(965)) - Recovering attempt: appattempt_1560421737895_0006_000001 with final state = NONE 2019-06-18 10:34:02,299 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1560421737895_0006_000001 2019-06-18 10:34:02,299 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1560421737895_0006_000001 2019-06-18 10:34:02,299 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1560421737895_0006 user: hive leaf-queue of parent: root #applications: 3 2019-06-18 10:34:02,311 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationOnRecovery(808)) - Accepted application application_1560421737895_0006 from user: hive, in queue: llap 2019-06-18 10:34:02,311 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0006_000001 State change from NEW to LAUNCHED on event = RECOVER 2019-06-18 10:34:02,312 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0006 State change from NEW to ACCEPTED on event = RECOVER 2019-06-18 10:34:02,312 INFO rmapp.RMAppImpl (RMAppImpl.java:recover(930)) - Recovering app: application_1560421737895_0007 with 1 attempts and final state = NONE 2019-06-18 10:34:02,313 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:recover(965)) - Recovering attempt: appattempt_1560421737895_0007_000001 with final state = NONE 2019-06-18 10:34:02,313 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1560421737895_0007_000001 2019-06-18 10:34:02,313 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1560421737895_0007_000001 2019-06-18 10:34:02,323 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1560421737895_0007 user: hive leaf-queue of parent: root #applications: 4 2019-06-18 10:34:02,323 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationOnRecovery(808)) - Accepted application application_1560421737895_0007 from user: hive, in queue: llap 2019-06-18 10:34:02,326 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0007_000001 State change from NEW to LAUNCHED on event = RECOVER 2019-06-18 10:34:02,326 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0007 State change from NEW to ACCEPTED on event = RECOVER 2019-06-18 10:34:02,330 INFO rmapp.RMAppImpl (RMAppImpl.java:recover(930)) - Recovering app: application_1560421737895_0008 with 1 attempts and final state = NONE 2019-06-18 10:34:02,331 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:recover(965)) - Recovering attempt: appattempt_1560421737895_0008_000001 with final state = NONE 2019-06-18 10:34:02,331 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1560421737895_0008_000001 2019-06-18 10:34:02,331 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1560421737895_0008_000001 2019-06-18 10:34:02,331 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1560421737895_0008 user: hive leaf-queue of parent: root #applications: 5 2019-06-18 10:34:02,332 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationOnRecovery(808)) - Accepted application application_1560421737895_0008 from user: hive, in queue: default 2019-06-18 10:34:02,332 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0008_000001 State change from NEW to LAUNCHED on event = RECOVER 2019-06-18 10:34:02,332 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0008 State change from NEW to ACCEPTED on event = RECOVER 2019-06-18 10:34:02,333 INFO rmapp.RMAppImpl (RMAppImpl.java:recover(930)) - Recovering app: application_1560421737895_0009 with 1 attempts and final state = NONE 2019-06-18 10:34:02,333 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:recover(965)) - Recovering attempt: appattempt_1560421737895_0009_000001 with final state = NONE 2019-06-18 10:34:02,333 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1560421737895_0009_000001 2019-06-18 10:34:02,333 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1560421737895_0009_000001 2019-06-18 10:34:02,333 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1560421737895_0009 user: hive leaf-queue of parent: root #applications: 6 2019-06-18 10:34:02,333 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationOnRecovery(808)) - Accepted application application_1560421737895_0009 from user: hive, in queue: llap 2019-06-18 10:34:02,334 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0009_000001 State change from NEW to LAUNCHED on event = RECOVER 2019-06-18 10:34:02,334 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0009 State change from NEW to ACCEPTED on event = RECOVER 2019-06-18 10:34:02,334 INFO rmapp.RMAppImpl (RMAppImpl.java:recover(930)) - Recovering app: application_1560421737895_0010 with 1 attempts and final state = NONE 2019-06-18 10:34:02,335 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:recover(965)) - Recovering attempt: appattempt_1560421737895_0010_000001 with final state = NONE 2019-06-18 10:34:02,335 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1560421737895_0010_000001 2019-06-18 10:34:02,335 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1560421737895_0010_000001 2019-06-18 10:34:02,335 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1560421737895_0010 user: hive leaf-queue of parent: root #applications: 7 2019-06-18 10:34:02,335 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationOnRecovery(808)) - Accepted application application_1560421737895_0010 from user: hive, in queue: llap 2019-06-18 10:34:02,336 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0010_000001 State change from NEW to LAUNCHED on event = RECOVER 2019-06-18 10:34:02,337 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0010 State change from NEW to ACCEPTED on event = RECOVER 2019-06-18 10:34:02,337 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 13 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,338 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 14 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,341 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 15 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,343 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 16 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,344 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 18 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,353 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=spark OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560421737895_0013 submitted by user spark to unknown queue: streaming APPID=application_1560421737895_0013 2019-06-18 10:34:02,353 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 19 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,356 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0014 2019-06-18 10:34:02,356 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 22 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,358 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 23 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,359 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 24 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,362 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 25 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,363 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 26 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,363 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0015 2019-06-18 10:34:02,365 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0016 2019-06-18 10:34:02,366 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0018 2019-06-18 10:34:02,366 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0019 2019-06-18 10:34:02,367 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0022 2019-06-18 10:34:02,368 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=spark OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560421737895_0023 submitted by user spark to unknown queue: streaming APPID=application_1560421737895_0023 2019-06-18 10:34:02,368 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0024 2019-06-18 10:34:02,369 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 27 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,371 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 28 is invalid, because it is out of the range [1, 5]. Use the global max attempts instead. 2019-06-18 10:34:02,373 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(564)) - Successfully recovered 40 out of 40 applications 2019-06-18 10:34:02,373 INFO resourcemanager.ResourceManager (ResourceManager.java:serviceStart(845)) - Recovery ended 2019-06-18 10:34:02,373 INFO security.RMContainerTokenSecretManager (RMContainerTokenSecretManager.java:rollMasterKey(109)) - Rolling master-key for container-tokens 2019-06-18 10:34:02,375 INFO security.NMTokenSecretManagerInRM (NMTokenSecretManagerInRM.java:rollMasterKey(95)) - Rolling master-key for nm-tokens 2019-06-18 10:34:02,376 INFO delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:updateCurrentKey(343)) - Updating the current master key for generating delegation tokens 2019-06-18 10:34:02,376 INFO security.RMDelegationTokenSecretManager (RMDelegationTokenSecretManager.java:storeNewMasterKey(92)) - storing master key with keyID 21 2019-06-18 10:34:02,376 INFO recovery.RMStateStore (RMStateStore.java:transition(498)) - Storing RMDTMasterKey. 2019-06-18 10:34:02,376 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:storeRMDTMasterKeyState(565)) - Storing RMDelegationKey_21 2019-06-18 10:34:02,377 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0025 2019-06-18 10:34:02,378 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0026 2019-06-18 10:34:02,379 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0027 2019-06-18 10:34:02,381 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(318)) - USER=spark OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1560421737895_0028 2019-06-18 10:34:02,527 INFO delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:run(675)) - Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) 2019-06-18 10:34:02,527 INFO delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:updateCurrentKey(343)) - Updating the current master key for generating delegation tokens 2019-06-18 10:34:02,528 INFO security.RMDelegationTokenSecretManager (RMDelegationTokenSecretManager.java:storeNewMasterKey(92)) - storing master key with keyID 22 2019-06-18 10:34:02,528 INFO recovery.RMStateStore (RMStateStore.java:transition(498)) - Storing RMDTMasterKey. 2019-06-18 10:34:02,528 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:storeRMDTMasterKeyState(565)) - Storing RMDelegationKey_22 2019-06-18 10:34:02,530 INFO event.AsyncDispatcher (AsyncDispatcher.java:register(223)) - Registering class org.apache.hadoop.yarn.nodelabels.event.NodeLabelsStoreEventType for class org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler 2019-06-18 10:34:02,547 INFO ipc.CallQueueManager (CallQueueManager.java:(84)) - Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 6400 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2019-06-18 10:34:02,565 INFO ipc.Server (Server.java:run(1070)) - Starting Socket Reader #1 for port 8025 2019-06-18 10:34:02,573 INFO pb.RpcServerFactoryPBImpl (RpcServerFactoryPBImpl.java:createServer(173)) - Adding protocol org.apache.hadoop.yarn.server.api.ResourceTrackerPB to the server 2019-06-18 10:34:02,576 INFO ipc.Server (Server.java:run(1310)) - IPC Server Responder: starting 2019-06-18 10:34:02,576 INFO ipc.Server (Server.java:run(1149)) - IPC Server listener on 8025: starting 2019-06-18 10:34:02,684 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Starting JVM pause monitor 2019-06-18 10:34:02,718 INFO ipc.CallQueueManager (CallQueueManager.java:(84)) - Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 6400 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2019-06-18 10:34:02,732 INFO ipc.Server (Server.java:run(1070)) - Starting Socket Reader #1 for port 8030 2019-06-18 10:34:02,745 INFO pb.RpcServerFactoryPBImpl (RpcServerFactoryPBImpl.java:createServer(173)) - Adding protocol org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB to the server 2019-06-18 10:34:02,745 INFO ipc.Server (Server.java:run(1310)) - IPC Server Responder: starting 2019-06-18 10:34:02,745 INFO ipc.Server (Server.java:run(1149)) - IPC Server listener on 8030: starting 2019-06-18 10:34:02,837 INFO ipc.CallQueueManager (CallQueueManager.java:(84)) - Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 6400 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2019-06-18 10:34:02,840 INFO ipc.Server (Server.java:run(1070)) - Starting Socket Reader #1 for port 8050 2019-06-18 10:34:02,845 INFO pb.RpcServerFactoryPBImpl (RpcServerFactoryPBImpl.java:createServer(173)) - Adding protocol org.apache.hadoop.yarn.api.ApplicationClientProtocolPB to the server 2019-06-18 10:34:02,845 INFO ipc.Server (Server.java:run(1310)) - IPC Server Responder: starting 2019-06-18 10:34:02,846 INFO ipc.Server (Server.java:run(1149)) - IPC Server listener on 8050: starting 2019-06-18 10:34:02,856 INFO ipc.Server (Server.java:saslProcess(1845)) - Auth successful for appattempt_1560421737895_0007_000001 (auth:SIMPLE) 2019-06-18 10:34:02,858 INFO client.SystemServiceManagerImpl (SystemServiceManagerImpl.java:scanForUserServices(281)) - Scan for launch type on /services 2019-06-18 10:34:02,861 WARN client.SystemServiceManagerImpl (SystemServiceManagerImpl.java:scanForUserServices(299)) - System service directory /services doesn't not exist. 2019-06-18 10:34:02,862 INFO resourcemanager.ResourceManager (ResourceManager.java:transitionToActive(1271)) - Transitioned to active state 2019-06-18 10:34:03,053 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-02.novalocal:45454 2019-06-18 10:34:03,053 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-01.novalocal:45454 2019-06-18 10:34:03,053 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-20.novalocal:45454 2019-06-18 10:34:03,056 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(403)) - Application attempt appattempt_1560421737895_0007_000001 doesn't exist in ApplicationMasterService cache. 2019-06-18 10:34:03,057 INFO ipc.Server (Server.java:logException(2722)) - IPC Server handler 0 on 8030, call Call#40071 Retry#0 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 10.0.96.35:36124 org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1560421737895_0007_000001 doesn't exist in ApplicationMasterService cache. at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:404) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) 2019-06-18 10:34:03,101 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-18.novalocal:45454 2019-06-18 10:34:03,106 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-17.novalocal:45454 2019-06-18 10:34:03,111 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-13.novalocal:45454 2019-06-18 10:34:03,114 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-09.novalocal:45454 2019-06-18 10:34:03,119 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-15.novalocal:45454 2019-06-18 10:34:03,123 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-03.novalocal:45454 2019-06-18 10:34:03,124 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-19.novalocal:45454 2019-06-18 10:34:03,126 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-16.novalocal:45454 2019-06-18 10:34:03,129 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-14.novalocal:45454 2019-06-18 10:34:03,130 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-05.novalocal:45454 2019-06-18 10:34:03,130 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-04.novalocal:45454 2019-06-18 10:34:03,138 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-10.novalocal:45454 2019-06-18 10:34:03,138 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-07.novalocal:45454 2019-06-18 10:34:03,178 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-12.novalocal:45454 2019-06-18 10:34:03,202 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-11.novalocal:45454 2019-06-18 10:34:03,208 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-08.novalocal:45454 2019-06-18 10:34:03,213 INFO ipc.Server (Server.java:saslProcess(1845)) - Auth successful for appattempt_1560421737895_0008_000001 (auth:SIMPLE) 2019-06-18 10:34:03,216 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(403)) - Application attempt appattempt_1560421737895_0008_000001 doesn't exist in ApplicationMasterService cache. 2019-06-18 10:34:03,216 INFO ipc.Server (Server.java:logException(2722)) - IPC Server handler 1 on 8030, call Call#40071 Retry#0 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 10.0.96.29:34194 org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1560421737895_0008_000001 doesn't exist in ApplicationMasterService cache. at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:404) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) 2019-06-18 10:34:03,239 INFO ipc.Server (Server.java:saslProcess(1845)) - Auth successful for appattempt_1560421737895_0005_000001 (auth:SIMPLE) 2019-06-18 10:34:03,241 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:unRegisterNodeManager(748)) - Node not found, ignoring the unregister from node id : ngbf134c6-task-instance-wcpl8ozo-06.novalocal:45454 2019-06-18 10:34:03,242 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(403)) - Application attempt appattempt_1560421737895_0005_000001 doesn't exist in ApplicationMasterService cache. 2019-06-18 10:34:03,242 INFO ipc.Server (Server.java:logException(2722)) - IPC Server handler 2 on 8030, call Call#40071 Retry#0 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 10.0.96.36:56922 org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1560421737895_0005_000001 doesn't exist in ApplicationMasterService cache. at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:404) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) 2019-06-18 10:34:03,376 INFO ipc.Server (Server.java:saslProcess(1845)) - Auth successful for appattempt_1560421737895_0004_000001 (auth:SIMPLE) 2019-06-18 10:34:03,380 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(403)) - Application attempt appattempt_1560421737895_0004_000001 doesn't exist in ApplicationMasterService cache. 2019-06-18 10:34:03,380 INFO ipc.Server (Server.java:logException(2722)) - IPC Server handler 3 on 8030, call Call#400452 Retry#0 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 10.0.96.35:36126 org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1560421737895_0004_000001 doesn't exist in ApplicationMasterService cache. at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:404) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) 2019-06-18 10:34:03,733 INFO ipc.Server (Server.java:saslProcess(1845)) - Auth successful for appattempt_1560421737895_0009_000001 (auth:SIMPLE) 2019-06-18 10:34:03,736 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(403)) - Application attempt appattempt_1560421737895_0009_000001 doesn't exist in ApplicationMasterService cache. 2019-06-18 10:34:03,736 INFO ipc.Server (Server.java:logException(2722)) - IPC Server handler 4 on 8030, call Call#40070 Retry#0 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 10.0.96.33:37176 org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1560421737895_0009_000001 doesn't exist in ApplicationMasterService cache. at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:404) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) 2019-06-18 10:34:03,807 INFO ipc.Server (Server.java:saslProcess(1845)) - Auth successful for appattempt_1560421737895_0006_000001 (auth:SIMPLE) 2019-06-18 10:34:03,810 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(403)) - Application attempt appattempt_1560421737895_0006_000001 doesn't exist in ApplicationMasterService cache. 2019-06-18 10:34:03,810 INFO ipc.Server (Server.java:logException(2722)) - IPC Server handler 0 on 8030, call Call#43645 Retry#0 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 10.0.96.26:46144 org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1560421737895_0006_000001 doesn't exist in ApplicationMasterService cache. at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:404) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) 2019-06-18 10:34:03,876 INFO ipc.Server (Server.java:saslProcess(1845)) - Auth successful for appattempt_1560421737895_0010_000001 (auth:SIMPLE) 2019-06-18 10:34:03,879 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(403)) - Application attempt appattempt_1560421737895_0010_000001 doesn't exist in ApplicationMasterService cache. 2019-06-18 10:34:03,879 INFO ipc.Server (Server.java:logException(2722)) - IPC Server handler 1 on 8030, call Call#40267 Retry#0 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 10.0.96.28:47514 org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1560421737895_0010_000001 doesn't exist in ApplicationMasterService cache. at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:404) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) 2019-06-18 10:34:07,742 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-20.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-20.novalocal:45454 2019-06-18 10:34:07,742 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-02.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-02.novalocal:45454 2019-06-18 10:34:07,746 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-20.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:07,746 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-02.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:07,770 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-20.novalocal:45454 clusterResource: 2019-06-18 10:34:07,771 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-02.novalocal:45454 clusterResource: 2019-06-18 10:34:07,824 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-01.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-01.novalocal:45454 2019-06-18 10:34:07,824 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-01.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:07,827 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-01.novalocal:45454 clusterResource: 2019-06-18 10:34:09,030 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-17.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-17.novalocal:45454 2019-06-18 10:34:09,030 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-17.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,032 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-17.novalocal:45454 clusterResource: 2019-06-18 10:34:09,051 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-18.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-18.novalocal:45454 2019-06-18 10:34:09,051 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-18.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,052 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-18.novalocal:45454 clusterResource: 2019-06-18 10:34:09,065 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-15.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-15.novalocal:45454 2019-06-18 10:34:09,065 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-15.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,066 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-15.novalocal:45454 clusterResource: 2019-06-18 10:34:09,107 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-13.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-13.novalocal:45454 2019-06-18 10:34:09,107 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-13.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,108 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-13.novalocal:45454 clusterResource: 2019-06-18 10:34:09,130 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-05.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-05.novalocal:45454 2019-06-18 10:34:09,131 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-05.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,132 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-05.novalocal:45454 clusterResource: 2019-06-18 10:34:09,151 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-16.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-16.novalocal:45454 2019-06-18 10:34:09,151 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-16.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,152 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-16.novalocal:45454 clusterResource: 2019-06-18 10:34:09,187 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-14.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-14.novalocal:45454 2019-06-18 10:34:09,187 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-14.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,188 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-14.novalocal:45454 clusterResource: 2019-06-18 10:34:09,257 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-19.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-19.novalocal:45454 2019-06-18 10:34:09,257 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-19.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,258 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-19.novalocal:45454 clusterResource: 2019-06-18 10:34:09,267 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-08.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-08.novalocal:45454 2019-06-18 10:34:09,267 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-08.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,268 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-08.novalocal:45454 clusterResource: 2019-06-18 10:34:09,301 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-06.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-06.novalocal:45454 2019-06-18 10:34:09,301 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-06.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,302 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-06.novalocal:45454 clusterResource: 2019-06-18 10:34:09,351 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-09.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-09.novalocal:45454 2019-06-18 10:34:09,351 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-09.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,353 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-09.novalocal:45454 clusterResource: 2019-06-18 10:34:09,357 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-10.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-10.novalocal:45454 2019-06-18 10:34:09,357 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-10.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,359 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-10.novalocal:45454 clusterResource: 2019-06-18 10:34:09,393 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-11.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-11.novalocal:45454 2019-06-18 10:34:09,393 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-11.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,395 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-11.novalocal:45454 clusterResource: 2019-06-18 10:34:09,457 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-03.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-03.novalocal:45454 2019-06-18 10:34:09,457 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-03.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,457 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-07.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-07.novalocal:45454 2019-06-18 10:34:09,457 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-07.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,460 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-03.novalocal:45454 clusterResource: 2019-06-18 10:34:09,461 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-07.novalocal:45454 clusterResource: 2019-06-18 10:34:09,482 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-04.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-04.novalocal:45454 2019-06-18 10:34:09,482 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-04.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,484 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-04.novalocal:45454 clusterResource: 2019-06-18 10:34:09,494 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-wcpl8ozo-12.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-wcpl8ozo-12.novalocal:45454 2019-06-18 10:34:09,494 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-wcpl8ozo-12.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:09,495 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-wcpl8ozo-12.novalocal:45454 clusterResource: 2019-06-18 10:34:10,698 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(335)) - USER=yarn OPERATION=Get Applications Request TARGET=ClientRMService RESULT=SUCCESS 2019-06-18 10:34:20,755 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(335)) - USER=yarn OPERATION=Get Applications Request TARGET=ClientRMService RESULT=SUCCESS 2019-06-18 10:34:24,062 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ng4959d89-core-instance-z7x6atpf-4.novalocal:45454 2019-06-18 10:34:24,063 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ng4959d89-core-instance-z7x6atpf-2.novalocal:45454 2019-06-18 10:34:24,063 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-1.novalocal:45454 2019-06-18 10:34:24,088 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-1.novalocal:45454 2019-06-18 10:34:24,092 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ng4959d89-core-instance-z7x6atpf-2.novalocal:45454 2019-06-18 10:34:24,101 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(460)) - received container statuses on node manager register :[[container_1560421737895_0005_01_000001, CreateTime: 1560424528077, Version: 0, State: COMPLETE, Capability: , Diagnostics: , ExitStatus: 0, NodeLabelExpression: , Priority: 0, AllocationRequestId: -1, AllocationTags: [], ]] 2019-06-18 10:34:24,102 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ng4959d89-core-instance-z7x6atpf-4.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:24,105 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ng4959d89-core-instance-z7x6atpf-4.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ng4959d89-core-instance-z7x6atpf-4.novalocal:45454 2019-06-18 10:34:24,107 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ng4959d89-core-instance-z7x6atpf-4.novalocal:45454 clusterResource: 2019-06-18 10:34:24,111 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1412)) - Updating application attempt appattempt_1560421737895_0005_000001 with final state: FAILED, and exit status: 0 2019-06-18 10:34:24,112 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0005_000001 State change from LAUNCHED to FINAL_SAVING on event = CONTAINER_FINISHED 2019-06-18 10:34:24,112 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationAttemptStateInternal(464)) - Updating info for attempt: appattempt_1560421737895_0005_000001 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0005/appattempt_1560421737895_0005_000001 2019-06-18 10:34:24,148 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(496)) - Unregistering app attempt : appattempt_1560421737895_0005_000001 2019-06-18 10:34:24,149 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1560421737895_0005_000001 2019-06-18 10:34:24,149 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0005_000001 State change from FINAL_SAVING to FAILED on event = ATTEMPT_UPDATE_SAVED 2019-06-18 10:34:24,150 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1534)) - Max app attempts is 1 for application_1560421737895_0005, preventing further attempts. 2019-06-18 10:34:24,151 INFO rmapp.RMAppImpl (RMAppImpl.java:rememberTargetTransitionsAndStoreState(1278)) - Updating application application_1560421737895_0005 with final state: FAILED 2019-06-18 10:34:24,154 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0005 State change from ACCEPTED to FINAL_SAVING on event = ATTEMPT_FAILED 2019-06-18 10:34:24,154 INFO recovery.RMStateStore (RMStateStore.java:transition(260)) - Updating info for app: application_1560421737895_0005 2019-06-18 10:34:24,154 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1048)) - Application Attempt appattempt_1560421737895_0005_000001 is done. finalState=FAILED 2019-06-18 10:34:24,154 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1056)) - Unknown application appattempt_1560421737895_0005_000001 has completed! 2019-06-18 10:34:24,157 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationStateInternal(423)) - Updating info for app: application_1560421737895_0005 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0005/application_1560421737895_0005 2019-06-18 10:34:24,192 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1197)) - Application application_1560421737895_0005 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560421737895_0005_000001 exited with exitCode: 0 Failing this attempt.Diagnostics: For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560421737895_0005 Then click on links to logs of each attempt. . Failing the application. 2019-06-18 10:34:24,194 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0005 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED 2019-06-18 10:34:24,194 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=hive OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560421737895_0005 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560421737895_0005_000001 exited with exitCode: 0 Failing this attempt.Diagnostics: For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560421737895_0005 Then click on links to logs of each attempt. . Failing the application. APPID=application_1560421737895_0005 2019-06-18 10:34:24,194 INFO capacity.ParentQueue (ParentQueue.java:removeApplication(522)) - Application removed - appId: application_1560421737895_0005 user: hive leaf-queue of parent: root #applications: 6 2019-06-18 10:34:24,209 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-4.novalocal:45454 2019-06-18 10:34:24,234 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-4.novalocal:45454 2019-06-18 10:34:24,263 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ng4959d89-core-instance-z7x6atpf-1.novalocal:45454 2019-06-18 10:34:24,267 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-5.novalocal:45454 2019-06-18 10:34:24,289 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-si6oryc7-5.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-si6oryc7-5.novalocal:45454 2019-06-18 10:34:24,289 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(460)) - received container statuses on node manager register :[[container_1560421737895_0008_01_000001, CreateTime: 1560424527905, Version: 0, State: COMPLETE, Capability: , Diagnostics: , ExitStatus: 0, NodeLabelExpression: , Priority: 0, AllocationRequestId: -1, AllocationTags: [], ]] 2019-06-18 10:34:24,289 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ng4959d89-core-instance-z7x6atpf-1.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:24,289 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ng4959d89-core-instance-z7x6atpf-1.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ng4959d89-core-instance-z7x6atpf-1.novalocal:45454 2019-06-18 10:34:24,290 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-si6oryc7-5.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:24,290 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1412)) - Updating application attempt appattempt_1560421737895_0008_000001 with final state: FAILED, and exit status: 0 2019-06-18 10:34:24,291 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0008_000001 State change from LAUNCHED to FINAL_SAVING on event = CONTAINER_FINISHED 2019-06-18 10:34:24,291 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationAttemptStateInternal(464)) - Updating info for attempt: appattempt_1560421737895_0008_000001 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0008/appattempt_1560421737895_0008_000001 2019-06-18 10:34:24,292 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ng4959d89-core-instance-z7x6atpf-1.novalocal:45454 clusterResource: 2019-06-18 10:34:24,293 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-si6oryc7-5.novalocal:45454 clusterResource: 2019-06-18 10:34:24,324 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(496)) - Unregistering app attempt : appattempt_1560421737895_0008_000001 2019-06-18 10:34:24,324 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1560421737895_0008_000001 2019-06-18 10:34:24,324 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0008_000001 State change from FINAL_SAVING to FAILED on event = ATTEMPT_UPDATE_SAVED 2019-06-18 10:34:24,324 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1534)) - Max app attempts is 1 for application_1560421737895_0008, preventing further attempts. 2019-06-18 10:34:24,324 INFO rmapp.RMAppImpl (RMAppImpl.java:rememberTargetTransitionsAndStoreState(1278)) - Updating application application_1560421737895_0008 with final state: FAILED 2019-06-18 10:34:24,325 INFO recovery.RMStateStore (RMStateStore.java:transition(260)) - Updating info for app: application_1560421737895_0008 2019-06-18 10:34:24,326 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0008 State change from ACCEPTED to FINAL_SAVING on event = ATTEMPT_FAILED 2019-06-18 10:34:24,327 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationStateInternal(423)) - Updating info for app: application_1560421737895_0008 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0008/application_1560421737895_0008 2019-06-18 10:34:24,327 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1048)) - Application Attempt appattempt_1560421737895_0008_000001 is done. finalState=FAILED 2019-06-18 10:34:24,327 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1056)) - Unknown application appattempt_1560421737895_0008_000001 has completed! 2019-06-18 10:34:24,363 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1197)) - Application application_1560421737895_0008 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560421737895_0008_000001 exited with exitCode: 0 Failing this attempt.Diagnostics: For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560421737895_0008 Then click on links to logs of each attempt. . Failing the application. 2019-06-18 10:34:24,364 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0008 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED 2019-06-18 10:34:24,364 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=hive OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560421737895_0008 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560421737895_0008_000001 exited with exitCode: 0 Failing this attempt.Diagnostics: For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560421737895_0008 Then click on links to logs of each attempt. . Failing the application. APPID=application_1560421737895_0008 2019-06-18 10:34:24,364 INFO capacity.ParentQueue (ParentQueue.java:removeApplication(522)) - Application removed - appId: application_1560421737895_0008 user: hive leaf-queue of parent: root #applications: 5 2019-06-18 10:34:24,386 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-3.novalocal:45454 2019-06-18 10:34:24,395 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ng4959d89-core-instance-z7x6atpf-5.novalocal:45454 2019-06-18 10:34:24,411 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-3.novalocal:45454 2019-06-18 10:34:24,429 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ng4959d89-core-instance-z7x6atpf-5.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:24,429 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(460)) - received container statuses on node manager register :[[container_1560421737895_0010_01_000001, CreateTime: 1560424528368, Version: 0, State: COMPLETE, Capability: , Diagnostics: , ExitStatus: 0, NodeLabelExpression: , Priority: 0, AllocationRequestId: -1, AllocationTags: [], ]] 2019-06-18 10:34:24,429 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ng4959d89-core-instance-z7x6atpf-5.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ng4959d89-core-instance-z7x6atpf-5.novalocal:45454 2019-06-18 10:34:24,430 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1412)) - Updating application attempt appattempt_1560421737895_0010_000001 with final state: FAILED, and exit status: 0 2019-06-18 10:34:24,431 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0010_000001 State change from LAUNCHED to FINAL_SAVING on event = CONTAINER_FINISHED 2019-06-18 10:34:24,431 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationAttemptStateInternal(464)) - Updating info for attempt: appattempt_1560421737895_0010_000001 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0010/appattempt_1560421737895_0010_000001 2019-06-18 10:34:24,432 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ng4959d89-core-instance-z7x6atpf-5.novalocal:45454 clusterResource: 2019-06-18 10:34:24,466 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(496)) - Unregistering app attempt : appattempt_1560421737895_0010_000001 2019-06-18 10:34:24,466 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1560421737895_0010_000001 2019-06-18 10:34:24,466 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0010_000001 State change from FINAL_SAVING to FAILED on event = ATTEMPT_UPDATE_SAVED 2019-06-18 10:34:24,466 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1534)) - Max app attempts is 1 for application_1560421737895_0010, preventing further attempts. 2019-06-18 10:34:24,466 INFO rmapp.RMAppImpl (RMAppImpl.java:rememberTargetTransitionsAndStoreState(1278)) - Updating application application_1560421737895_0010 with final state: FAILED 2019-06-18 10:34:24,467 INFO recovery.RMStateStore (RMStateStore.java:transition(260)) - Updating info for app: application_1560421737895_0010 2019-06-18 10:34:24,467 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0010 State change from ACCEPTED to FINAL_SAVING on event = ATTEMPT_FAILED 2019-06-18 10:34:24,469 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationStateInternal(423)) - Updating info for app: application_1560421737895_0010 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0010/application_1560421737895_0010 2019-06-18 10:34:24,469 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1048)) - Application Attempt appattempt_1560421737895_0010_000001 is done. finalState=FAILED 2019-06-18 10:34:24,469 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1056)) - Unknown application appattempt_1560421737895_0010_000001 has completed! 2019-06-18 10:34:24,505 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1197)) - Application application_1560421737895_0010 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560421737895_0010_000001 exited with exitCode: 0 Failing this attempt.Diagnostics: For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560421737895_0010 Then click on links to logs of each attempt. . Failing the application. 2019-06-18 10:34:24,506 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0010 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED 2019-06-18 10:34:24,506 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=hive OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560421737895_0010 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560421737895_0010_000001 exited with exitCode: 0 Failing this attempt.Diagnostics: For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560421737895_0010 Then click on links to logs of each attempt. . Failing the application. APPID=application_1560421737895_0010 2019-06-18 10:34:24,506 INFO capacity.ParentQueue (ParentQueue.java:removeApplication(522)) - Application removed - appId: application_1560421737895_0010 user: hive leaf-queue of parent: root #applications: 4 2019-06-18 10:34:24,837 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-2.novalocal:45454 2019-06-18 10:34:24,864 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ngbf134c6-task-instance-si6oryc7-2.novalocal:45454 2019-06-18 10:34:24,907 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(536)) - Node not found resyncing ng4959d89-core-instance-z7x6atpf-3.novalocal:45454 2019-06-18 10:34:24,930 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ng4959d89-core-instance-z7x6atpf-3.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ng4959d89-core-instance-z7x6atpf-3.novalocal:45454 2019-06-18 10:34:24,930 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ng4959d89-core-instance-z7x6atpf-3.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:24,932 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ng4959d89-core-instance-z7x6atpf-3.novalocal:45454 clusterResource: 2019-06-18 10:34:25,087 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ngbf134c6-task-instance-si6oryc7-1.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:25,087 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(460)) - received container statuses on node manager register :[[container_1560421737895_0004_01_000002, CreateTime: 1560424468469, Version: 0, State: COMPLETE, Capability: , Diagnostics: [2019-06-18 10:34:24.064]Container Killed by ResourceManager [2019-06-18 10:34:24.076]Container killed on request. Exit code is 143 [2019-06-18 10:34:24.086]Container exited with a non-zero exit code 143. , ExitStatus: -106, NodeLabelExpression: , Priority: 0, AllocationRequestId: 0, AllocationTags: [], ]] 2019-06-18 10:34:25,087 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-si6oryc7-1.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-si6oryc7-1.novalocal:45454 2019-06-18 10:34:25,088 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ngbf134c6-task-instance-si6oryc7-1.novalocal:45454 clusterResource: 2019-06-18 10:34:25,095 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(671)) - ng4959d89-core-instance-z7x6atpf-2.novalocal:45454 Node Transitioned from NEW to RUNNING 2019-06-18 10:34:25,095 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(460)) - received container statuses on node manager register :[[container_1560421737895_0004_01_000001, CreateTime: 1560424464231, Version: 0, State: COMPLETE, Capability: , Diagnostics: [2019-06-18 10:34:24.067]Container Killed by ResourceManager [2019-06-18 10:34:24.080]Container killed on request. Exit code is 143 [2019-06-18 10:34:24.089]Container exited with a non-zero exit code 143. , ExitStatus: -106, NodeLabelExpression: , Priority: 0, AllocationRequestId: -1, AllocationTags: [], ], [container_1560421737895_0007_01_000001, CreateTime: 1560424527989, Version: 0, State: COMPLETE, Capability: , Diagnostics: , ExitStatus: 0, NodeLabelExpression: , Priority: 0, AllocationRequestId: -1, AllocationTags: [], ]] 2019-06-18 10:34:25,095 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ng4959d89-core-instance-z7x6atpf-2.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ng4959d89-core-instance-z7x6atpf-2.novalocal:45454 2019-06-18 10:34:25,096 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1412)) - Updating application attempt appattempt_1560421737895_0004_000001 with final state: FAILED, and exit status: -106 2019-06-18 10:34:25,097 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0004_000001 State change from LAUNCHED to FINAL_SAVING on event = CONTAINER_FINISHED 2019-06-18 10:34:25,097 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationAttemptStateInternal(464)) - Updating info for attempt: appattempt_1560421737895_0004_000001 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0004/appattempt_1560421737895_0004_000001 2019-06-18 10:34:25,098 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1412)) - Updating application attempt appattempt_1560421737895_0007_000001 with final state: FAILED, and exit status: 0 2019-06-18 10:34:25,098 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1865)) - Added node ng4959d89-core-instance-z7x6atpf-2.novalocal:45454 clusterResource: 2019-06-18 10:34:25,098 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0007_000001 State change from LAUNCHED to FINAL_SAVING on event = CONTAINER_FINISHED 2019-06-18 10:34:25,139 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(496)) - Unregistering app attempt : appattempt_1560421737895_0004_000001 2019-06-18 10:34:25,139 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationAttemptStateInternal(464)) - Updating info for attempt: appattempt_1560421737895_0007_000001 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0007/appattempt_1560421737895_0007_000001 2019-06-18 10:34:25,139 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1560421737895_0004_000001 2019-06-18 10:34:25,139 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0004_000001 State change from FINAL_SAVING to FAILED on event = ATTEMPT_UPDATE_SAVED 2019-06-18 10:34:25,139 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1538)) - The number of failed attempts is 0. The max attempts is 5 2019-06-18 10:34:25,142 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1048)) - Application Attempt appattempt_1560421737895_0004_000001 is done. finalState=FAILED 2019-06-18 10:34:25,142 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1056)) - Unknown application appattempt_1560421737895_0004_000001 has completed! 2019-06-18 10:34:25,174 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:registerAppAttempt(479)) - Registering app attempt : appattempt_1560421737895_0004_000002 2019-06-18 10:34:25,175 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0004_000002 State change from NEW to SUBMITTED on event = START 2019-06-18 10:34:25,178 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(496)) - Unregistering app attempt : appattempt_1560421737895_0007_000001 2019-06-18 10:34:25,178 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1560421737895_0007_000001 2019-06-18 10:34:25,178 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1560421737895_0007_000001 State change from FINAL_SAVING to FAILED on event = ATTEMPT_UPDATE_SAVED 2019-06-18 10:34:25,178 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1534)) - Max app attempts is 1 for application_1560421737895_0007, preventing further attempts. 2019-06-18 10:34:25,178 INFO rmapp.RMAppImpl (RMAppImpl.java:rememberTargetTransitionsAndStoreState(1278)) - Updating application application_1560421737895_0007 with final state: FAILED 2019-06-18 10:34:25,180 INFO recovery.RMStateStore (RMStateStore.java:transition(260)) - Updating info for app: application_1560421737895_0007 2019-06-18 10:34:25,180 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0007 State change from ACCEPTED to FINAL_SAVING on event = ATTEMPT_FAILED 2019-06-18 10:34:25,181 INFO recovery.FileSystemRMStateStore (FileSystemRMStateStore.java:updateApplicationStateInternal(423)) - Updating info for app: application_1560421737895_0007 at: /tmp/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1560421737895_0007/application_1560421737895_0007 2019-06-18 10:34:25,192 FATAL event.EventDispatcher (EventDispatcher.java:run(75)) - Error in handling event type APP_ATTEMPT_ADDED to the Event Dispatcher java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.transferStateFromPreviousAttempt(SchedulerApplicationAttempt.java:1158) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.transferStateFromPreviousAttempt(FiCaSchedulerApp.java:852) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:982) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1730) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:167) at org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66) at java.lang.Thread.run(Thread.java:748) 2019-06-18 10:34:25,193 INFO event.EventDispatcher (EventDispatcher.java:run(79)) - Exiting, bbye.. 2019-06-18 10:34:25,197 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.w.WebAppContext@2047981{/,null,UNAVAILABLE}{/cluster} 2019-06-18 10:34:25,201 INFO server.AbstractConnector (AbstractConnector.java:doStop(318)) - Stopped ServerConnector@776015fc{HTTP/1.1,[http/1.1]}{0.0.0.0:8088} 2019-06-18 10:34:25,202 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.w.WebAppContext@6f31df32{/ui2,null,UNAVAILABLE} 2019-06-18 10:34:25,202 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.s.ServletContextHandler@2bebb74f{/static,jar:file:/opt/share/hadoop/yarn/hadoop-yarn-common-3.1.1.jar!/webapps/static,UNAVAILABLE} 2019-06-18 10:34:25,202 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.s.ServletContextHandler@28501a4b{/logs,file:///opt/log/hadoop-yarn/yarn/,UNAVAILABLE} 2019-06-18 10:34:25,206 INFO ipc.Server (Server.java:stop(3077)) - Stopping server on 8050 2019-06-18 10:34:25,210 INFO ipc.Server (Server.java:run(1181)) - Stopping IPC Server listener on 8050 2019-06-18 10:34:25,211 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,211 INFO ipc.Server (Server.java:run(1315)) - Stopping IPC Server Responder 2019-06-18 10:34:25,211 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,211 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,212 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,212 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,212 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,213 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,213 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,214 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,215 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:25,220 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1197)) - Application application_1560421737895_0007 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560421737895_0007_000001 exited with exitCode: 0 Failing this attempt.Diagnostics: For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560421737895_0007 Then click on links to logs of each attempt. . Failing the application. 2019-06-18 10:34:25,221 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1560421737895_0007 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED 2019-06-18 10:34:25,221 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(478)) - USER=hive OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1560421737895_0007 failed 1 times (global limit =5; local limit is =1) due to Attempt recovered after RM restartAM Container for appattempt_1560421737895_0007_000001 exited with exitCode: 0 Failing this attempt.Diagnostics: For more detailed output, check the application tracking page: http://ng02e8517-master-instance-4zau3mvg.novalocal:8088/cluster/app/application_1560421737895_0007 Then click on links to logs of each attempt. . Failing the application. APPID=application_1560421737895_0007 2019-06-18 10:34:25,221 INFO ipc.Server (Server.java:stop(3077)) - Stopping server on 8141 2019-06-18 10:34:25,222 INFO ipc.Server (Server.java:run(1181)) - Stopping IPC Server listener on 8141 2019-06-18 10:34:25,223 INFO ipc.Server (Server.java:run(1315)) - Stopping IPC Server Responder 2019-06-18 10:34:25,223 INFO resourcemanager.ResourceManager (ResourceManager.java:transitionToStandby(1282)) - Transitioning to standby state 2019-06-18 10:34:25,224 INFO client.SystemServiceManagerImpl (SystemServiceManagerImpl.java:serviceStop(137)) - Stopping org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl 2019-06-18 10:34:25,224 WARN amlauncher.ApplicationMasterLauncher (ApplicationMasterLauncher.java:run(122)) - org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning. 2019-06-18 10:34:25,224 INFO ipc.Server (Server.java:stop(3077)) - Stopping server on 8030 2019-06-18 10:34:25,227 INFO ipc.Server (Server.java:run(1181)) - Stopping IPC Server listener on 8030 2019-06-18 10:34:25,228 INFO ipc.Server (Server.java:run(1315)) - Stopping IPC Server Responder 2019-06-18 10:34:25,229 INFO ipc.Server (Server.java:stop(3077)) - Stopping server on 8025 2019-06-18 10:34:25,233 INFO ipc.Server (Server.java:doRead(1252)) - Socket Reader #1 for port 8025: readAndProcess from client 10.0.96.56:49694 threw exception [java.nio.channels.ClosedByInterruptException] java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:407) at org.apache.hadoop.ipc.Server.channelRead(Server.java:3265) at org.apache.hadoop.ipc.Server.access$2600(Server.java:137) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2040) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1245) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1101) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1072) 2019-06-18 10:34:25,233 INFO ipc.Server (Server.java:run(1181)) - Stopping IPC Server listener on 8025 2019-06-18 10:34:25,233 INFO util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(156)) - NMLivelinessMonitor thread interrupted 2019-06-18 10:34:25,237 INFO ipc.Server (Server.java:run(1315)) - Stopping IPC Server Responder 2019-06-18 10:34:25,241 WARN net.ScriptBasedMapping (ScriptBasedMapping.java:runResolveCommand(254)) - Exception running /etc/hadoop/conf/topology_script.py 10.0.96.33 java.io.InterruptedIOException: java.lang.InterruptedException at org.apache.hadoop.util.Shell.runCommand(Shell.java:1012) at org.apache.hadoop.util.Shell.run(Shell.java:902) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227) at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251) at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188) at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119) at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101) at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:95) at org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.resolve(ResourceTrackerService.java:831) at org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.registerNodeManager(ResourceTrackerService.java:398) at org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceTrackerPBServiceImpl.registerNodeManager(ResourceTrackerPBServiceImpl.java:59) at org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$2.callBlockingMethod(ResourceTracker.java:95) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395) at org.apache.hadoop.util.Shell.runCommand(Shell.java:1002) ... 19 more 2019-06-18 10:34:25,242 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(460)) - received container statuses on node manager register :[[container_1560421737895_0004_01_000003, CreateTime: 1560424468471, Version: 0, State: COMPLETE, Capability: , Diagnostics: [2019-06-18 10:34:24.210]Container Killed by ResourceManager [2019-06-18 10:34:24.223]Container killed on request. Exit code is 143 [2019-06-18 10:34:24.232]Container exited with a non-zero exit code 143. , ExitStatus: -106, NodeLabelExpression: , Priority: 0, AllocationRequestId: 0, AllocationTags: [], ], [container_1560421737895_0009_01_000001, CreateTime: 1560424528033, Version: 0, State: COMPLETE, Capability: , Diagnostics: , ExitStatus: 0, NodeLabelExpression: , Priority: 0, AllocationRequestId: -1, AllocationTags: [], ]] 2019-06-18 10:34:25,242 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(495)) - NodeManager from node ngbf134c6-task-instance-si6oryc7-4.novalocal(cmPort: 45454 httpPort: 8042) registered with capability: , assigned nodeId ngbf134c6-task-instance-si6oryc7-4.novalocal:45454 2019-06-18 10:34:25,242 WARN ipc.Server (Server.java:processResponse(1519)) - IPC Server handler 39 on 8025, call Call#425394 Retry#0 org.apache.hadoop.yarn.server.api.ResourceTrackerPB.registerNodeManager from 10.0.96.33:42216: output error 2019-06-18 10:34:25,242 INFO ipc.Server (Server.java:run(2691)) - IPC Server handler 39 on 8025 caught an exception java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461) at org.apache.hadoop.ipc.Server.channelWrite(Server.java:3245) at org.apache.hadoop.ipc.Server.access$1700(Server.java:137) at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:1469) at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1539) at org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2589) at org.apache.hadoop.ipc.Server$Connection.access$300(Server.java:1611) at org.apache.hadoop.ipc.Server$RpcCall.doResponse(Server.java:936) at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:770) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:881) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) 2019-06-18 10:34:35,194 WARN util.ShutdownHookManager (ShutdownHookManager.java:run(71)) - ShutdownHook 'CompositeServiceShutdownHook' timeout, java.util.concurrent.TimeoutException java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask.get(FutureTask.java:205) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:68) 2019-06-18 10:34:35,195 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service SchedulerEventDispatcher failed in state STOPPED org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:107) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:867) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1230) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1286) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1338) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:105) ... 20 more 2019-06-18 10:34:35,196 WARN service.CompositeService (ServiceOperations.java:stopQuietly(104)) - When stopping the service SchedulerEventDispatcher org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:107) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:867) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1230) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1286) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1338) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:105) ... 20 more 2019-06-18 10:34:35,196 INFO capacity.CapacityScheduler (CapacityScheduler.java:run(609)) - AsyncScheduleThread[Thread-12] exited! 2019-06-18 10:34:35,196 ERROR capacity.CapacityScheduler (CapacityScheduler.java:run(647)) - java.lang.InterruptedException 2019-06-18 10:34:35,196 INFO capacity.CapacityScheduler (CapacityScheduler.java:run(651)) - ResourceCommitterService exited! 2019-06-18 10:34:35,198 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events. 2019-06-18 10:34:35,198 INFO util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(156)) - org.apache.hadoop.yarn.server.resourcemanager.rmapp.monitor.RMAppLifetimeMonitor thread interrupted 2019-06-18 10:34:35,198 INFO util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(156)) - AMLivelinessMonitor thread interrupted 2019-06-18 10:34:35,198 INFO util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(156)) - org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted 2019-06-18 10:34:35,198 INFO util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(156)) - AMLivelinessMonitor thread interrupted 2019-06-18 10:34:35,199 ERROR delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:run(696)) - ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2019-06-18 10:34:35,200 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service RMActiveServices failed in state STOPPED org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:107) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:867) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1230) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1286) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1338) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:105) ... 20 more 2019-06-18 10:34:35,200 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service ResourceManager failed in state STOPPED org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:107) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:867) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1230) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1286) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1338) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:105) ... 20 more 2019-06-18 10:34:35,200 WARN service.AbstractService (ServiceOperations.java:stopQuietly(104)) - When stopping the service ResourceManager org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:107) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:867) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1230) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1286) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1338) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.yarn.event.EventDispatcher.serviceStop(EventDispatcher.java:105) ... 20 more 2019-06-18 10:34:35,201 INFO resourcemanager.ResourceManager (LogAdapter.java:info(49)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ResourceManager at ng02e8517-master-instance-4zau3mvg.novalocal/10.0.96.30