2017-09-03 19:33:21,256 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e26_1504267919597_6793_01_000122 transitioned from RUNNING to KILLING 2017-09-03 19:33:21,256 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e26_1504267919597_6793_01_000141 transitioned from RUNNING to KILLING 2017-09-03 19:33:21,256 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e26_1504267919597_6793_01_000122 2017-09-03 19:33:21,256 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: Using container runtime: DefaultLinuxContainerRuntime 2017-09-03 19:33:21,261 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor: Shell execution returned exit code: 143. Privileged Execution Operation Output: main : command provided 1 main : run as user is hdfs main : requested yarn user is hdfs Getting exit code file... Creating script paths... Writing pid file... Writing to tmp file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1504267919597_6793/container_e26_1504267919597_6793_01_000122/container_e26_1504267919597_6793_01_000122.pid.tmp Writing to cgroup task files... Creating local dirs... Launching container... Getting exit code file... Creating script paths... Full command array for failed execution: [nice, -n, 0, /usr/lib/hadoop-yarn/bin/container-executor, hdfs, hdfs, 1, application_1504267919597_6793, container_e26_1504267919597_6793_01_000122, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1504267919597_6793/container_e26_1504267919597_6793_01_000122, /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1504267919597_6793/container_e26_1504267919597_6793_01_000122/launch_container.sh, /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1504267919597_6793/container_e26_1504267919597_6793_01_000122/container_e26_1504267919597_6793_01_000122.tokens, /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1504267919597_6793/container_e26_1504267919597_6793_01_000122/container_e26_1504267919597_6793_01_000122.pid, /tmp/hadoop-yarn/nm-local-dir, /var/log/hadoop-yarn/userlogs, cgroups=/cgroup/cpu/yarn/container_e26_1504267919597_6793_01_000122/tasks] 2017-09-03 19:33:21,261 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime: Launch container failed. Exception: org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=143: at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:177) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:103) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:90) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:384) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: ExitCodeException exitCode=143: at org.apache.hadoop.util.Shell.runCommand(Shell.java:972) at org.apache.hadoop.util.Shell.run(Shell.java:869) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:151) ... 9 more 2017-09-03 19:33:21,261 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container container_e26_1504267919597_6793_01_000122 is : 143 2017-09-03 19:33:21,270 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e26_1504267919597_6793_01_000141 2017-09-03 19:33:21,270 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: Using container runtime: DefaultLinuxContainerRuntime 2017-09-03 19:33:21,274 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor: Shell execution returned exit code: 143. Privileged Execution Operation Output: main : command provided 1 main : run as user is hdfs main : requested yarn user is hdfs Getting exit code file... Creating script paths... Writing pid file... Writing to tmp file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1504267919597_6793/container_e26_1504267919597_6793_01_000141/container_e26_1504267919597_6793_01_000141.pid.tmp Writing to cgroup task files... Creating local dirs... Launching container... Getting exit code file... Creating script paths... Full command array for failed execution: [nice, -n, 0, /usr/lib/hadoop-yarn/bin/container-executor, hdfs, hdfs, 1, application_1504267919597_6793, container_e26_1504267919597_6793_01_000141, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1504267919597_6793/container_e26_1504267919597_6793_01_000141, /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1504267919597_6793/container_e26_1504267919597_6793_01_000141/launch_container.sh, /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1504267919597_6793/container_e26_1504267919597_6793_01_000141/container_e26_1504267919597_6793_01_000141.tokens, /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1504267919597_6793/container_e26_1504267919597_6793_01_000141/container_e26_1504267919597_6793_01_000141.pid, /tmp/hadoop-yarn/nm-local-dir, /var/log/hadoop-yarn/userlogs, cgroups=/cgroup/cpu/yarn/container_e26_1504267919597_6793_01_000141/tasks] 2017-09-03 19:33:21,274 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime: Launch container failed. Exception: org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=143: at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:177) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:103) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:90) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:384) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: ExitCodeException exitCode=143: at org.apache.hadoop.util.Shell.runCommand(Shell.java:972) at org.apache.hadoop.util.Shell.run(Shell.java:869) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:151) ... 9 more 2017-09-03 19:33:21,274 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container container_e26_1504267919597_6793_01_000141 is : 143 2017-09-03 19:33:21,510 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: Using container runtime: DefaultLinuxContainerRuntime 2017-09-03 19:33:21,524 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: Using container runtime: DefaultLinuxContainerRuntime 2017-09-03 19:33:21,674 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e26_1504267919597_6793_01_000122 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2017-09-03 19:33:21,674 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1504267919597_6793/container_e26_1504267919597_6793_01_000122 2017-09-03 19:33:21,674 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1504267919597_6793 CONTAINERID=container_e26_1504267919597_6793_01_000122 2017-09-03 19:33:21,674 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e26_1504267919597_6793_01_000122 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2017-09-03 19:33:21,674 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e26_1504267919597_6793_01_000122 from application application_1504267919597_6793 2017-09-03 19:33:21,674 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e26_1504267919597_6793_01_000122 2017-09-03 19:33:21,674 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1504267919597_6793 2017-09-03 19:33:21,674 INFO org.apache.spark.network.yarn.YarnShuffleService: Stopping container container_e26_1504267919597_6793_01_000122 2017-09-03 19:33:21,675 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e26_1504267919597_6793_01_000122]