2016-11-02 18:17:37,191 INFO org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /tmp/hadoop-yarn/nm-local-dir passed disk check, adding to list of valid directories. 2016-11-02 18:17:37,214 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) turned good: 1/1 local-dirs are good: /tmp/hadoop-yarn/nm-local-dir; 1/1 log-dirs are good: /var/log/hadoop-yarn/userlogs 2016-11-02 18:17:39,184 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0109_000001 (auth:SIMPLE) 2016-11-02 18:17:39,622 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0110_000001 (auth:SIMPLE) 2016-11-02 18:17:39,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0109_01_000001 by user hdfs 2016-11-02 18:17:39,718 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0109 2016-11-02 18:17:39,737 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0109 transitioned from NEW to INITING 2016-11-02 18:17:39,738 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0109_01_000001 to application application_1478007721435_0109 2016-11-02 18:17:39,749 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0109 CONTAINERID=container_e06_1478007721435_0109_01_000001 2016-11-02 18:17:39,760 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0110_01_000001 by user hdfs 2016-11-02 18:17:39,779 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0110 2016-11-02 18:17:39,780 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0110 CONTAINERID=container_e06_1478007721435_0110_01_000001 2016-11-02 18:17:40,317 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:17:40,415 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0110 transitioned from NEW to INITING 2016-11-02 18:17:40,415 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0110_01_000001 to application application_1478007721435_0110 2016-11-02 18:17:40,472 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0109 transitioned from INITING to RUNNING 2016-11-02 18:17:40,475 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0109_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:17:40,476 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0110 transitioned from INITING to RUNNING 2016-11-02 18:17:40,476 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0109 2016-11-02 18:17:40,497 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0110_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:17:40,497 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0109_01_000001 2016-11-02 18:17:40,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0110 2016-11-02 18:17:40,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0110_01_000001 2016-11-02 18:17:40,512 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0109_01_000001.tokens. Credentials list: 2016-11-02 18:17:40,513 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0110_01_000001.tokens. Credentials list: 2016-11-02 18:17:40,520 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:17:40,520 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:17:40,532 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0110_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0110/container_e06_1478007721435_0110_01_000001.tokens 2016-11-02 18:17:40,532 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0109_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0109/container_e06_1478007721435_0109_01_000001.tokens 2016-11-02 18:17:40,533 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0110 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0110 2016-11-02 18:17:40,533 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0109 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0109 2016-11-02 18:17:40,634 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0111_000001 (auth:SIMPLE) 2016-11-02 18:17:40,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0111_01_000001 by user hdfs 2016-11-02 18:17:40,649 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0111 2016-11-02 18:17:40,652 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0111 transitioned from NEW to INITING 2016-11-02 18:17:40,656 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:17:40,664 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0111 CONTAINERID=container_e06_1478007721435_0111_01_000001 2016-11-02 18:17:40,730 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0111_01_000001 to application application_1478007721435_0111 2016-11-02 18:17:40,731 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0111 transitioned from INITING to RUNNING 2016-11-02 18:17:40,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0111_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:17:40,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0111 2016-11-02 18:17:40,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0111_01_000001 2016-11-02 18:17:40,737 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0111_01_000001.tokens. Credentials list: 2016-11-02 18:17:40,738 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:17:40,740 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0111_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0111/container_e06_1478007721435_0111_01_000001.tokens 2016-11-02 18:17:40,741 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0111 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0111 2016-11-02 18:17:41,642 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0112_000001 (auth:SIMPLE) 2016-11-02 18:17:41,647 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0112_01_000001 by user hdfs 2016-11-02 18:17:41,649 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0112 2016-11-02 18:17:41,652 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0112 transitioned from NEW to INITING 2016-11-02 18:17:41,652 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0112_01_000001 to application application_1478007721435_0112 2016-11-02 18:17:41,652 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0112 CONTAINERID=container_e06_1478007721435_0112_01_000001 2016-11-02 18:17:41,711 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0112 transitioned from INITING to RUNNING 2016-11-02 18:17:41,712 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0112_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:17:41,712 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0112 2016-11-02 18:17:41,713 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0112_01_000001 2016-11-02 18:17:41,716 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0112_01_000001.tokens. Credentials list: 2016-11-02 18:17:41,717 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:17:41,722 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0112_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0112/container_e06_1478007721435_0112_01_000001.tokens 2016-11-02 18:17:41,722 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0112 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0112 2016-11-02 18:17:42,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0109_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:17:42,608 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0109_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:17:42,614 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0109_01_000001 2016-11-02 18:17:42,620 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0109/container_e06_1478007721435_0109_01_000001/default_container_executor.sh] 2016-11-02 18:17:42,622 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0111_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:17:42,643 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0113_000001 (auth:SIMPLE) 2016-11-02 18:17:42,647 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0110_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:17:42,651 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0113_01_000001 by user hdfs 2016-11-02 18:17:42,657 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0113 2016-11-02 18:17:42,658 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0113 transitioned from NEW to INITING 2016-11-02 18:17:42,658 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0113_01_000001 to application application_1478007721435_0113 2016-11-02 18:17:42,658 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0113 CONTAINERID=container_e06_1478007721435_0113_01_000001 2016-11-02 18:17:42,661 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:17:42,697 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0111/container_e06_1478007721435_0111_01_000001/default_container_executor.sh] 2016-11-02 18:17:42,713 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0110/container_e06_1478007721435_0110_01_000001/default_container_executor.sh] 2016-11-02 18:17:42,741 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0111_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:17:42,741 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0110_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:17:42,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0111_01_000001 2016-11-02 18:17:42,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0110_01_000001 2016-11-02 18:17:42,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0113 transitioned from INITING to RUNNING 2016-11-02 18:17:42,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0113_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:17:42,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0113 2016-11-02 18:17:42,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0113_01_000001 2016-11-02 18:17:42,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0113_01_000001.tokens. Credentials list: 2016-11-02 18:17:42,746 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:17:42,751 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0113_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0113/container_e06_1478007721435_0113_01_000001.tokens 2016-11-02 18:17:42,752 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0113 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0113 2016-11-02 18:17:43,455 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0112_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:17:43,510 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0109_01_000001 since CPU usage is not yet available. 2016-11-02 18:17:43,559 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0112_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:17:43,559 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0112_01_000001 2016-11-02 18:17:43,569 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0112/container_e06_1478007721435_0112_01_000001/default_container_executor.sh] 2016-11-02 18:17:43,581 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0110_01_000001 since CPU usage is not yet available. 2016-11-02 18:17:43,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0111_01_000001 since CPU usage is not yet available. 2016-11-02 18:17:43,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0112_01_000001 since CPU usage is not yet available. 2016-11-02 18:17:45,740 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0113_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:17:46,357 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0113_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:17:46,357 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0113_01_000001 2016-11-02 18:17:46,365 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0113/container_e06_1478007721435_0113_01_000001/default_container_executor.sh] 2016-11-02 18:17:47,549 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0113_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:03,914 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0111_01_000001 is : 15 2016-11-02 18:18:03,918 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0111_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0111_01_000001 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:03,920 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:03,921 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:04,398 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0109_01_000001 is : 15 2016-11-02 18:18:04,398 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0109_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:04,400 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:04,400 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0109_01_000001 2016-11-02 18:18:04,401 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:04,401 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:04,401 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:04,401 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:04,406 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:04,408 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:04,408 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:04,408 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:04,408 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:04,408 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:04,408 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:04,408 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:04,408 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:04,409 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:04,419 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0111_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:04,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0111_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:04,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0109_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:04,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0111_01_000001 2016-11-02 18:18:04,474 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0109_01_000001 2016-11-02 18:18:04,476 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0111/container_e06_1478007721435_0111_01_000001 2016-11-02 18:18:04,497 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0109/container_e06_1478007721435_0109_01_000001 2016-11-02 18:18:04,498 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0111 CONTAINERID=container_e06_1478007721435_0111_01_000001 2016-11-02 18:18:04,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0111_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:04,501 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0109 CONTAINERID=container_e06_1478007721435_0109_01_000001 2016-11-02 18:18:04,502 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0109_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:04,502 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0111_01_000001 from application application_1478007721435_0111 2016-11-02 18:18:04,502 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0111_01_000001 2016-11-02 18:18:04,503 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0111_01_000001 for log-aggregation 2016-11-02 18:18:04,503 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0111 2016-11-02 18:18:04,504 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0109_01_000001 from application application_1478007721435_0109 2016-11-02 18:18:04,504 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0109_01_000001 2016-11-02 18:18:04,504 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0109_01_000001 for log-aggregation 2016-11-02 18:18:04,504 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0109 2016-11-02 18:18:05,184 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0110_01_000001 is : 15 2016-11-02 18:18:05,184 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0110_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0110_01_000001 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:05,185 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:05,185 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:05,188 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0110_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:05,188 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0110_01_000001 2016-11-02 18:18:05,213 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0110/container_e06_1478007721435_0110_01_000001 2016-11-02 18:18:05,213 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0110 CONTAINERID=container_e06_1478007721435_0110_01_000001 2016-11-02 18:18:05,215 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0110_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:05,215 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0110_01_000001 from application application_1478007721435_0110 2016-11-02 18:18:05,216 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0110_01_000001 2016-11-02 18:18:05,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0110_01_000001 for log-aggregation 2016-11-02 18:18:05,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0110 2016-11-02 18:18:05,225 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0111_01_000001] 2016-11-02 18:18:05,227 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0111 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:05,228 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0111 2016-11-02 18:18:05,228 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0111 2016-11-02 18:18:05,232 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0111 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:05,232 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0111 2016-11-02 18:18:05,244 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0100_000002 (auth:SIMPLE) 2016-11-02 18:18:05,251 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0100_02_000001 by user hdfs 2016-11-02 18:18:05,254 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0100 2016-11-02 18:18:05,255 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0100 CONTAINERID=container_e06_1478007721435_0100_02_000001 2016-11-02 18:18:05,258 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0100 transitioned from NEW to INITING 2016-11-02 18:18:05,258 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0100_02_000001 to application application_1478007721435_0100 2016-11-02 18:18:05,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0100 transitioned from INITING to RUNNING 2016-11-02 18:18:05,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0100_02_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:05,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0100 2016-11-02 18:18:05,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0100_02_000001 2016-11-02 18:18:05,297 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0100_02_000001.tokens. Credentials list: 2016-11-02 18:18:05,297 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:05,300 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0100_02_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0100/container_e06_1478007721435_0100_02_000001.tokens 2016-11-02 18:18:05,300 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0100 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0100 2016-11-02 18:18:05,320 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0111_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:05,349 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0111/container_e06_1478007721435_0111_01_000001/stderr 2016-11-02 18:18:05,349 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0111/container_e06_1478007721435_0111_01_000001/stdout 2016-11-02 18:18:05,430 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0111 2016-11-02 18:18:06,087 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0112_01_000001 is : 15 2016-11-02 18:18:06,088 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0112_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0112_01_000001 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:06,088 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:06,092 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:06,100 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0112_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:06,100 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0112_01_000001 2016-11-02 18:18:06,124 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0112/container_e06_1478007721435_0112_01_000001 2016-11-02 18:18:06,126 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0112 CONTAINERID=container_e06_1478007721435_0112_01_000001 2016-11-02 18:18:06,126 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0112_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:06,126 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0112_01_000001 from application application_1478007721435_0112 2016-11-02 18:18:06,126 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0112_01_000001 2016-11-02 18:18:06,127 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0112_01_000001 for log-aggregation 2016-11-02 18:18:06,127 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0112 2016-11-02 18:18:06,130 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0113_01_000001 is : 15 2016-11-02 18:18:06,130 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0113_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:06,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:06,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0113_01_000001 2016-11-02 18:18:06,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0109_01_000001, container_e06_1478007721435_0110_01_000001] 2016-11-02 18:18:06,131 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0109 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0110 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:06,131 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0109 2016-11-02 18:18:06,132 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0109 2016-11-02 18:18:06,133 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0110 2016-11-02 18:18:06,135 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0110 2016-11-02 18:18:06,135 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0109 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:06,135 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0113_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:06,136 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0110 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:06,136 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0109 2016-11-02 18:18:06,136 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0113_01_000001 2016-11-02 18:18:06,147 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0108_000002 (auth:SIMPLE) 2016-11-02 18:18:06,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0108_02_000001 by user hdfs 2016-11-02 18:18:06,155 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0108 2016-11-02 18:18:06,158 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0108 CONTAINERID=container_e06_1478007721435_0108_02_000001 2016-11-02 18:18:06,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0109_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:06,172 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0109/container_e06_1478007721435_0109_01_000001/stdout 2016-11-02 18:18:06,173 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0109/container_e06_1478007721435_0109_01_000001/stderr 2016-11-02 18:18:06,181 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0113/container_e06_1478007721435_0113_01_000001 2016-11-02 18:18:06,181 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0110 2016-11-02 18:18:06,182 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0108 transitioned from NEW to INITING 2016-11-02 18:18:06,182 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0108_02_000001 to application application_1478007721435_0108 2016-11-02 18:18:06,182 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0113 CONTAINERID=container_e06_1478007721435_0113_01_000001 2016-11-02 18:18:06,183 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0113_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:06,186 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:06,188 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0112_01_000001] 2016-11-02 18:18:06,195 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0109 2016-11-02 18:18:06,204 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0104_000002 (auth:SIMPLE) 2016-11-02 18:18:06,216 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0104_02_000001 by user hdfs 2016-11-02 18:18:06,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0110_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:06,219 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0104 2016-11-02 18:18:06,220 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0113_01_000001 from application application_1478007721435_0113 2016-11-02 18:18:06,220 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0113_01_000001 2016-11-02 18:18:06,220 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0113_01_000001 for log-aggregation 2016-11-02 18:18:06,220 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0110/container_e06_1478007721435_0110_01_000001/stderr 2016-11-02 18:18:06,220 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0113 2016-11-02 18:18:06,220 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0112 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:06,221 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0110/container_e06_1478007721435_0110_01_000001/stdout 2016-11-02 18:18:06,221 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0104 transitioned from NEW to INITING 2016-11-02 18:18:06,221 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0104_02_000001 to application application_1478007721435_0104 2016-11-02 18:18:06,222 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0104 CONTAINERID=container_e06_1478007721435_0104_02_000001 2016-11-02 18:18:06,223 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0112 2016-11-02 18:18:06,223 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0112 2016-11-02 18:18:06,241 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0110 2016-11-02 18:18:06,265 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0112 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:06,266 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0108 transitioned from INITING to RUNNING 2016-11-02 18:18:06,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0112 2016-11-02 18:18:06,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0108_02_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:06,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0104 transitioned from INITING to RUNNING 2016-11-02 18:18:06,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0108 2016-11-02 18:18:06,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0104_02_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:06,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0108_02_000001 2016-11-02 18:18:06,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0104 2016-11-02 18:18:06,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0104_02_000001 2016-11-02 18:18:06,271 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0108_02_000001.tokens. Credentials list: 2016-11-02 18:18:06,271 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:06,273 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0108_02_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0108/container_e06_1478007721435_0108_02_000001.tokens 2016-11-02 18:18:06,273 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0108 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0108 2016-11-02 18:18:06,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0104_02_000001.tokens. Credentials list: 2016-11-02 18:18:06,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0100_02_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:06,274 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:06,277 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0104_02_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0104/container_e06_1478007721435_0104_02_000001.tokens 2016-11-02 18:18:06,277 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0104 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0104 2016-11-02 18:18:06,292 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0112_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:06,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0112/container_e06_1478007721435_0112_01_000001/stdout 2016-11-02 18:18:06,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0112/container_e06_1478007721435_0112_01_000001/stderr 2016-11-02 18:18:06,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0100_02_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:06,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0100_02_000001 2016-11-02 18:18:06,311 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0100/container_e06_1478007721435_0100_02_000001/default_container_executor.sh] 2016-11-02 18:18:06,317 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0112 2016-11-02 18:18:06,327 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: { hdfs://root:9820/user/hdfs/.sparkStaging/application_1478007721435_0108/__spark_conf__530730564444367028.zip, 1478078052870, ARCHIVE, null } failed: File does not exist: hdfs://root:9820/user/hdfs/.sparkStaging/application_1478007721435_0108/__spark_conf__530730564444367028.zip java.io.FileNotFoundException: File does not exist: hdfs://root:9820/user/hdfs/.sparkStaging/application_1478007721435_0108/__spark_conf__530730564444367028.zip at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1370) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1363) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1378) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:06,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0108_02_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:18:06,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Container container_e06_1478007721435_0108_02_000001 sent RELEASE event on a resource request { hdfs://root:9820/user/hdfs/.sparkStaging/application_1478007721435_0108/__spark_conf__530730564444367028.zip, 1478078052870, ARCHIVE, null } not present in cache. 2016-11-02 18:18:06,330 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Unknown localizer with localizerId container_e06_1478007721435_0108_02_000001 is sending heartbeat. Ordering it to DIE 2016-11-02 18:18:06,331 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0108 CONTAINERID=container_e06_1478007721435_0108_02_000001 2016-11-02 18:18:06,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0108_02_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:18:06,332 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0108_02_000001 from application application_1478007721435_0108 2016-11-02 18:18:06,332 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0108_02_000001 2016-11-02 18:18:06,332 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0108_02_000001 for log-aggregation 2016-11-02 18:18:06,335 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0108 2016-11-02 18:18:06,335 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed java.io.IOException: java.io.InterruptedIOException: Call interrupted at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:178) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:140) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1143) Caused by: java.io.InterruptedIOException: Call interrupted at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489) at org.apache.hadoop.ipc.Client.call(Client.java:1441) at org.apache.hadoop.ipc.Client.call(Client.java:1351) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy80.heartbeat(Unknown Source) at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:63) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:258) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:175) ... 2 more 2016-11-02 18:18:06,336 WARN org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher thread interrupted java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339) at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:254) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1167) 2016-11-02 18:18:06,337 ERROR org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[LocalizerRunner for container_e06_1478007721435_0108_02_000001,5,main] threw an Exception. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:262) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1167) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339) at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:254) ... 1 more 2016-11-02 18:18:06,338 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0113_01_000001] 2016-11-02 18:18:06,338 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0113 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:06,338 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0113 2016-11-02 18:18:06,339 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0113 2016-11-02 18:18:06,339 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0113 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:06,340 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0113 2016-11-02 18:18:06,353 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0106_000002 (auth:SIMPLE) 2016-11-02 18:18:06,360 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0113_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:06,360 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0113/container_e06_1478007721435_0113_01_000001/stderr 2016-11-02 18:18:06,360 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0106_02_000001 by user hdfs 2016-11-02 18:18:06,361 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0113/container_e06_1478007721435_0113_01_000001/stdout 2016-11-02 18:18:06,363 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0106 2016-11-02 18:18:06,364 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0106 CONTAINERID=container_e06_1478007721435_0106_02_000001 2016-11-02 18:18:06,365 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0106 transitioned from NEW to INITING 2016-11-02 18:18:06,365 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0106_02_000001 to application application_1478007721435_0106 2016-11-02 18:18:06,366 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:06,384 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0113 2016-11-02 18:18:06,416 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0106 transitioned from INITING to RUNNING 2016-11-02 18:18:06,416 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0106_02_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:06,416 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0106 2016-11-02 18:18:06,417 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0106_02_000001 2016-11-02 18:18:06,418 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0106_02_000001.tokens. Credentials list: 2016-11-02 18:18:06,418 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:06,422 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0106_02_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0106/container_e06_1478007721435_0106_02_000001.tokens 2016-11-02 18:18:06,423 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0106 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0106 2016-11-02 18:18:06,459 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: { hdfs://root:9820/user/hdfs/.sparkStaging/application_1478007721435_0106/__spark_conf__5617696307326660471.zip, 1478078046227, ARCHIVE, null } failed: File does not exist: hdfs://root:9820/user/hdfs/.sparkStaging/application_1478007721435_0106/__spark_conf__5617696307326660471.zip java.io.FileNotFoundException: File does not exist: hdfs://root:9820/user/hdfs/.sparkStaging/application_1478007721435_0106/__spark_conf__5617696307326660471.zip at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1370) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1363) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1378) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:06,460 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0106_02_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:18:06,460 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Container container_e06_1478007721435_0106_02_000001 sent RELEASE event on a resource request { hdfs://root:9820/user/hdfs/.sparkStaging/application_1478007721435_0106/__spark_conf__5617696307326660471.zip, 1478078046227, ARCHIVE, null } not present in cache. 2016-11-02 18:18:06,460 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Unknown localizer with localizerId container_e06_1478007721435_0106_02_000001 is sending heartbeat. Ordering it to DIE 2016-11-02 18:18:06,461 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0106 CONTAINERID=container_e06_1478007721435_0106_02_000001 2016-11-02 18:18:06,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed java.io.IOException: java.io.InterruptedIOException: Call interrupted at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:178) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:140) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1143) Caused by: java.io.InterruptedIOException: Call interrupted at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489) at org.apache.hadoop.ipc.Client.call(Client.java:1441) at org.apache.hadoop.ipc.Client.call(Client.java:1351) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy80.heartbeat(Unknown Source) at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:63) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:258) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:175) ... 2 more 2016-11-02 18:18:06,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0106_02_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:18:06,462 WARN org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher thread interrupted java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339) at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:254) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1167) 2016-11-02 18:18:06,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0106_02_000001 from application application_1478007721435_0106 2016-11-02 18:18:06,463 ERROR org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[LocalizerRunner for container_e06_1478007721435_0106_02_000001,5,main] threw an Exception. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:262) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1167) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339) at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:254) ... 1 more 2016-11-02 18:18:06,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0106_02_000001 2016-11-02 18:18:06,464 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0106_02_000001 for log-aggregation 2016-11-02 18:18:06,464 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0106 2016-11-02 18:18:06,465 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0108_02_000001] 2016-11-02 18:18:06,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0108 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:06,466 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0108 2016-11-02 18:18:06,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0108 2016-11-02 18:18:06,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0108 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:06,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0108 2016-11-02 18:18:06,483 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0108_02_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:06,484 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0114_000001 (auth:SIMPLE) 2016-11-02 18:18:06,488 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0114_01_000001 by user hdfs 2016-11-02 18:18:06,490 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0114 2016-11-02 18:18:06,491 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0114 transitioned from NEW to INITING 2016-11-02 18:18:06,491 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0114_01_000001 to application application_1478007721435_0114 2016-11-02 18:18:06,491 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0114 CONTAINERID=container_e06_1478007721435_0114_01_000001 2016-11-02 18:18:06,507 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0108 2016-11-02 18:18:06,543 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0114 transitioned from INITING to RUNNING 2016-11-02 18:18:06,544 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0114_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:06,544 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0114 2016-11-02 18:18:06,544 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0114_01_000001 2016-11-02 18:18:06,546 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0114_01_000001.tokens. Credentials list: 2016-11-02 18:18:06,546 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:06,547 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0114_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0114/container_e06_1478007721435_0114_01_000001.tokens 2016-11-02 18:18:06,548 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0114 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0114 2016-11-02 18:18:07,261 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0104_02_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:07,288 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0104_02_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:07,288 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0104_02_000001 2016-11-02 18:18:07,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0104/container_e06_1478007721435_0104_02_000001/default_container_executor.sh] 2016-11-02 18:18:07,335 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0114_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:07,359 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0114_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:07,359 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0114_01_000001 2016-11-02 18:18:07,364 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0114/container_e06_1478007721435_0114_01_000001/default_container_executor.sh] 2016-11-02 18:18:07,467 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0106_02_000001] 2016-11-02 18:18:07,467 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0114_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:07,468 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0106 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:07,469 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0106 2016-11-02 18:18:07,469 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0106 2016-11-02 18:18:07,469 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0106 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:07,469 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0106 2016-11-02 18:18:07,482 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0115_000001 (auth:SIMPLE) 2016-11-02 18:18:07,486 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0106_02_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:07,488 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0115_01_000001 by user hdfs 2016-11-02 18:18:07,491 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0115 2016-11-02 18:18:07,492 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0115 transitioned from NEW to INITING 2016-11-02 18:18:07,492 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0115_01_000001 to application application_1478007721435_0115 2016-11-02 18:18:07,494 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:07,495 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0115 CONTAINERID=container_e06_1478007721435_0115_01_000001 2016-11-02 18:18:07,509 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0106 2016-11-02 18:18:07,513 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0104_02_000001 since CPU usage is not yet available. 2016-11-02 18:18:07,535 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0115 transitioned from INITING to RUNNING 2016-11-02 18:18:07,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0115_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:07,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0115 2016-11-02 18:18:07,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0115_01_000001 2016-11-02 18:18:07,539 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0115_01_000001.tokens. Credentials list: 2016-11-02 18:18:07,540 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:07,541 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0115_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0115/container_e06_1478007721435_0115_01_000001.tokens 2016-11-02 18:18:07,541 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0115 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0115 2016-11-02 18:18:07,546 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0100_02_000001 since CPU usage is not yet available. 2016-11-02 18:18:08,492 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0116_000001 (auth:SIMPLE) 2016-11-02 18:18:08,494 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0115_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:08,499 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0116_01_000001 by user hdfs 2016-11-02 18:18:08,502 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0116 2016-11-02 18:18:08,502 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0116 transitioned from NEW to INITING 2016-11-02 18:18:08,502 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0116_01_000001 to application application_1478007721435_0116 2016-11-02 18:18:08,502 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0116 CONTAINERID=container_e06_1478007721435_0116_01_000001 2016-11-02 18:18:08,526 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0115/container_e06_1478007721435_0115_01_000001/default_container_executor.sh] 2016-11-02 18:18:08,549 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0115_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:08,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0115_01_000001 2016-11-02 18:18:08,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0116 transitioned from INITING to RUNNING 2016-11-02 18:18:08,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0116_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:08,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0116 2016-11-02 18:18:08,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0116_01_000001 2016-11-02 18:18:08,552 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0116_01_000001.tokens. Credentials list: 2016-11-02 18:18:08,552 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:08,558 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0116_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0116/container_e06_1478007721435_0116_01_000001.tokens 2016-11-02 18:18:08,558 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0116 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0116 2016-11-02 18:18:09,757 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0116_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:09,780 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0116_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:09,780 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0116_01_000001 2016-11-02 18:18:09,783 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0116/container_e06_1478007721435_0116_01_000001/default_container_executor.sh] 2016-11-02 18:18:10,708 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0116_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:10,771 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0115_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:23,326 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0100_02_000001 is : 15 2016-11-02 18:18:23,326 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0100_02_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0100_02_000001 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:23,327 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:23,327 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:23,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0100_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:23,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0100_02_000001 2016-11-02 18:18:23,360 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0100/container_e06_1478007721435_0100_02_000001 2016-11-02 18:18:23,360 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0100 CONTAINERID=container_e06_1478007721435_0100_02_000001 2016-11-02 18:18:23,363 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0100_02_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:23,363 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0100_02_000001 from application application_1478007721435_0100 2016-11-02 18:18:23,364 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0100_02_000001 2016-11-02 18:18:23,365 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0100_02_000001 for log-aggregation 2016-11-02 18:18:23,366 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0100 2016-11-02 18:18:24,371 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0100_02_000001] 2016-11-02 18:18:24,371 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0100 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:24,373 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0100 2016-11-02 18:18:24,373 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0100 2016-11-02 18:18:24,374 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0100 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:24,374 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0100 2016-11-02 18:18:24,385 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0117_000001 (auth:SIMPLE) 2016-11-02 18:18:24,390 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0117_01_000001 by user hdfs 2016-11-02 18:18:24,393 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0117 2016-11-02 18:18:24,393 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0117 transitioned from NEW to INITING 2016-11-02 18:18:24,393 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0117_01_000001 to application application_1478007721435_0117 2016-11-02 18:18:24,395 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0117 CONTAINERID=container_e06_1478007721435_0117_01_000001 2016-11-02 18:18:24,402 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:24,411 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0100_02_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:24,481 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0100/container_e06_1478007721435_0100_02_000001/stderr 2016-11-02 18:18:24,487 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0100/container_e06_1478007721435_0100_02_000001/stdout 2016-11-02 18:18:24,531 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0100 2016-11-02 18:18:24,535 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0117 transitioned from INITING to RUNNING 2016-11-02 18:18:24,536 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0117_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:24,536 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0117 2016-11-02 18:18:24,537 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0117_01_000001 2016-11-02 18:18:24,542 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0117_01_000001.tokens. Credentials list: 2016-11-02 18:18:24,542 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:24,544 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0117_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0117/container_e06_1478007721435_0117_01_000001.tokens 2016-11-02 18:18:24,544 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0117 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0117 2016-11-02 18:18:25,787 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0117_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:25,811 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0117_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:25,811 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0117_01_000001 2016-11-02 18:18:25,815 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0117/container_e06_1478007721435_0117_01_000001/default_container_executor.sh] 2016-11-02 18:18:25,842 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0114_01_000001 is : 15 2016-11-02 18:18:25,842 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0114_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0114_01_000001 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:25,842 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:25,843 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:25,844 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0114_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:25,844 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0114_01_000001 2016-11-02 18:18:25,865 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0114/container_e06_1478007721435_0114_01_000001 2016-11-02 18:18:25,865 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0114 CONTAINERID=container_e06_1478007721435_0114_01_000001 2016-11-02 18:18:25,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0114_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:25,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0114_01_000001 from application application_1478007721435_0114 2016-11-02 18:18:25,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0114_01_000001 2016-11-02 18:18:25,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0114_01_000001 for log-aggregation 2016-11-02 18:18:25,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0114 2016-11-02 18:18:25,887 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0104_02_000001 is : 15 2016-11-02 18:18:25,887 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0104_02_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0104_02_000001 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:25,887 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:25,887 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:25,889 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0104_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:25,889 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0104_02_000001 2016-11-02 18:18:25,909 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0104/container_e06_1478007721435_0104_02_000001 2016-11-02 18:18:25,909 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0104 CONTAINERID=container_e06_1478007721435_0104_02_000001 2016-11-02 18:18:25,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0104_02_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:25,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0104_02_000001 from application application_1478007721435_0104 2016-11-02 18:18:25,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0104_02_000001 2016-11-02 18:18:25,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0104_02_000001 for log-aggregation 2016-11-02 18:18:25,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0104 2016-11-02 18:18:25,913 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0114_01_000001] 2016-11-02 18:18:25,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0114 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:25,914 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0114 2016-11-02 18:18:25,914 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0114 2016-11-02 18:18:25,914 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0114 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:25,914 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0114 2016-11-02 18:18:25,927 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0118_000001 (auth:SIMPLE) 2016-11-02 18:18:25,932 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0114_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:25,933 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0114/container_e06_1478007721435_0114_01_000001/stdout 2016-11-02 18:18:25,933 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0118_01_000001 by user hdfs 2016-11-02 18:18:25,935 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0114/container_e06_1478007721435_0114_01_000001/stderr 2016-11-02 18:18:25,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0118 2016-11-02 18:18:25,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0118 transitioned from NEW to INITING 2016-11-02 18:18:25,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0118_01_000001 to application application_1478007721435_0118 2016-11-02 18:18:25,937 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0118 CONTAINERID=container_e06_1478007721435_0118_01_000001 2016-11-02 18:18:25,953 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0114 2016-11-02 18:18:25,970 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0118 transitioned from INITING to RUNNING 2016-11-02 18:18:25,970 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0118_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:25,970 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0118 2016-11-02 18:18:25,970 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0118_01_000001 2016-11-02 18:18:25,971 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0118_01_000001.tokens. Credentials list: 2016-11-02 18:18:25,972 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:25,973 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0118_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0118/container_e06_1478007721435_0118_01_000001.tokens 2016-11-02 18:18:25,973 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0118 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0118 2016-11-02 18:18:26,510 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0117_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:26,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0118_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:26,832 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0118_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:26,832 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0118_01_000001 2016-11-02 18:18:26,837 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0118/container_e06_1478007721435_0118_01_000001/default_container_executor.sh] 2016-11-02 18:18:26,920 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0104_02_000001] 2016-11-02 18:18:26,921 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0104 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:26,922 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0104 2016-11-02 18:18:26,922 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0104 2016-11-02 18:18:26,923 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0104 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:26,923 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0104 2016-11-02 18:18:26,931 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0119_000001 (auth:SIMPLE) 2016-11-02 18:18:26,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0119_01_000001 by user hdfs 2016-11-02 18:18:26,940 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0119 2016-11-02 18:18:26,941 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0119 transitioned from NEW to INITING 2016-11-02 18:18:26,941 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0119_01_000001 to application application_1478007721435_0119 2016-11-02 18:18:26,941 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0119 CONTAINERID=container_e06_1478007721435_0119_01_000001 2016-11-02 18:18:26,946 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:26,954 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0104_02_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:26,958 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0104/container_e06_1478007721435_0104_02_000001/stdout 2016-11-02 18:18:26,958 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0104/container_e06_1478007721435_0104_02_000001/stderr 2016-11-02 18:18:26,998 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0104 2016-11-02 18:18:27,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0119 transitioned from INITING to RUNNING 2016-11-02 18:18:27,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0119_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:27,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0119 2016-11-02 18:18:27,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0119_01_000001 2016-11-02 18:18:27,016 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0119_01_000001.tokens. Credentials list: 2016-11-02 18:18:27,016 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:27,017 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0119_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0119/container_e06_1478007721435_0119_01_000001.tokens 2016-11-02 18:18:27,017 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0119 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0119 2016-11-02 18:18:28,364 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0119_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:28,406 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0119_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:28,406 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0119_01_000001 2016-11-02 18:18:28,414 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0119/container_e06_1478007721435_0119_01_000001/default_container_executor.sh] 2016-11-02 18:18:29,612 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0118_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:29,751 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0119_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:30,677 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0116_01_000001 is : 15 2016-11-02 18:18:30,677 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0116_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0116_01_000001 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:30,677 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:30,677 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:30,680 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0116_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:30,680 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0116_01_000001 2016-11-02 18:18:30,716 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0116/container_e06_1478007721435_0116_01_000001 2016-11-02 18:18:30,716 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0116 CONTAINERID=container_e06_1478007721435_0116_01_000001 2016-11-02 18:18:30,720 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0116_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:30,720 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0116_01_000001 from application application_1478007721435_0116 2016-11-02 18:18:30,720 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0116_01_000001 2016-11-02 18:18:30,721 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0116_01_000001 for log-aggregation 2016-11-02 18:18:30,721 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0116 2016-11-02 18:18:31,715 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0115_01_000001 is : 15 2016-11-02 18:18:31,716 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0115_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0115_01_000001 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:31,716 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:31,717 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:31,719 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0115_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:31,719 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0115_01_000001 2016-11-02 18:18:31,725 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0116_01_000001] 2016-11-02 18:18:31,743 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0120_000001 (auth:SIMPLE) 2016-11-02 18:18:31,752 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0115/container_e06_1478007721435_0115_01_000001 2016-11-02 18:18:31,752 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0116 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:31,752 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0115 CONTAINERID=container_e06_1478007721435_0115_01_000001 2016-11-02 18:18:31,753 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0115_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:31,753 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0116 2016-11-02 18:18:31,753 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0116 2016-11-02 18:18:31,753 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0115_01_000001 from application application_1478007721435_0115 2016-11-02 18:18:31,753 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0115_01_000001 2016-11-02 18:18:31,754 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0115_01_000001 for log-aggregation 2016-11-02 18:18:31,754 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0115 2016-11-02 18:18:31,754 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0116 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:31,754 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0116 2016-11-02 18:18:31,771 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0120_01_000001 by user hdfs 2016-11-02 18:18:31,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0116_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:31,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0120 2016-11-02 18:18:31,778 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0120 transitioned from NEW to INITING 2016-11-02 18:18:31,778 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0120_01_000001 to application application_1478007721435_0120 2016-11-02 18:18:31,778 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0120 CONTAINERID=container_e06_1478007721435_0120_01_000001 2016-11-02 18:18:31,778 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0116/container_e06_1478007721435_0116_01_000001/stderr 2016-11-02 18:18:31,779 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0116/container_e06_1478007721435_0116_01_000001/stdout 2016-11-02 18:18:31,806 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0116 2016-11-02 18:18:31,830 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0120 transitioned from INITING to RUNNING 2016-11-02 18:18:31,831 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0120_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:31,831 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0120 2016-11-02 18:18:31,831 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0120_01_000001 2016-11-02 18:18:31,832 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0120_01_000001.tokens. Credentials list: 2016-11-02 18:18:31,832 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:31,834 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0120_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0120/container_e06_1478007721435_0120_01_000001.tokens 2016-11-02 18:18:31,834 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0120 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0120 2016-11-02 18:18:32,753 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0120_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:32,756 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0115_01_000001] 2016-11-02 18:18:32,756 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0115 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:32,757 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0115 2016-11-02 18:18:32,757 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0115 2016-11-02 18:18:32,757 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0115 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:32,757 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0115 2016-11-02 18:18:32,770 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0121_000001 (auth:SIMPLE) 2016-11-02 18:18:32,775 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0121_01_000001 by user hdfs 2016-11-02 18:18:32,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0121 2016-11-02 18:18:32,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0121 transitioned from NEW to INITING 2016-11-02 18:18:32,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0121_01_000001 to application application_1478007721435_0121 2016-11-02 18:18:32,778 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0121 CONTAINERID=container_e06_1478007721435_0121_01_000001 2016-11-02 18:18:32,778 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:32,807 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0115_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:32,813 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0115/container_e06_1478007721435_0115_01_000001/stdout 2016-11-02 18:18:32,816 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0115/container_e06_1478007721435_0115_01_000001/stderr 2016-11-02 18:18:32,833 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0120/container_e06_1478007721435_0120_01_000001/default_container_executor.sh] 2016-11-02 18:18:32,840 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0115 2016-11-02 18:18:32,861 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0120_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:32,862 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0120_01_000001 2016-11-02 18:18:32,862 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0121 transitioned from INITING to RUNNING 2016-11-02 18:18:32,862 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0121_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:32,862 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0121 2016-11-02 18:18:32,862 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0121_01_000001 2016-11-02 18:18:32,864 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0121_01_000001.tokens. Credentials list: 2016-11-02 18:18:32,864 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:32,865 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0121_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0121/container_e06_1478007721435_0121_01_000001.tokens 2016-11-02 18:18:32,866 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0121 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0121 2016-11-02 18:18:33,474 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0120_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:34,454 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0121_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:34,490 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0121_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:34,490 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0121_01_000001 2016-11-02 18:18:34,493 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0121/container_e06_1478007721435_0121_01_000001/default_container_executor.sh] 2016-11-02 18:18:36,529 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0121_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:39,491 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0117_01_000001 is : 15 2016-11-02 18:18:39,491 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0117_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0117_01_000001 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:39,492 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:39,492 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:39,494 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0117_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:39,494 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0117_01_000001 2016-11-02 18:18:39,531 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0117/container_e06_1478007721435_0117_01_000001 2016-11-02 18:18:39,531 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0117 CONTAINERID=container_e06_1478007721435_0117_01_000001 2016-11-02 18:18:39,532 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0117_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:39,532 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0117_01_000001 from application application_1478007721435_0117 2016-11-02 18:18:39,532 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0117_01_000001 2016-11-02 18:18:39,532 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0117_01_000001 for log-aggregation 2016-11-02 18:18:39,532 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0117 2016-11-02 18:18:40,537 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0117_01_000001] 2016-11-02 18:18:40,537 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0117 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:40,538 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0117 2016-11-02 18:18:40,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0117 2016-11-02 18:18:40,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0117 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:40,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0117 2016-11-02 18:18:40,551 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0122_000001 (auth:SIMPLE) 2016-11-02 18:18:40,562 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0122_01_000001 by user hdfs 2016-11-02 18:18:40,563 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0117_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:40,564 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0117/container_e06_1478007721435_0117_01_000001/stdout 2016-11-02 18:18:40,564 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0122 2016-11-02 18:18:40,564 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0117/container_e06_1478007721435_0117_01_000001/stderr 2016-11-02 18:18:40,565 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0122 transitioned from NEW to INITING 2016-11-02 18:18:40,565 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0122_01_000001 to application application_1478007721435_0122 2016-11-02 18:18:40,565 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0122 CONTAINERID=container_e06_1478007721435_0122_01_000001 2016-11-02 18:18:40,593 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0117 2016-11-02 18:18:40,615 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0122 transitioned from INITING to RUNNING 2016-11-02 18:18:40,615 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0122_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:40,615 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0122 2016-11-02 18:18:40,615 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0122_01_000001 2016-11-02 18:18:40,617 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0122_01_000001.tokens. Credentials list: 2016-11-02 18:18:40,618 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:40,620 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0122_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0122/container_e06_1478007721435_0122_01_000001.tokens 2016-11-02 18:18:40,620 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0122 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0122 2016-11-02 18:18:41,577 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0122_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:41,619 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0122_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:41,619 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0122_01_000001 2016-11-02 18:18:41,622 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0122/container_e06_1478007721435_0122_01_000001/default_container_executor.sh] 2016-11-02 18:18:42,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0122_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:43,130 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0118_01_000001 is : 15 2016-11-02 18:18:43,130 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0118_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0118_01_000001 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:43,130 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:43,131 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:43,132 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0118_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:43,132 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0118_01_000001 2016-11-02 18:18:43,157 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0118/container_e06_1478007721435_0118_01_000001 2016-11-02 18:18:43,158 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0118 CONTAINERID=container_e06_1478007721435_0118_01_000001 2016-11-02 18:18:43,159 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0118_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:43,159 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0118_01_000001 from application application_1478007721435_0118 2016-11-02 18:18:43,159 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0118_01_000001 2016-11-02 18:18:43,159 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0118_01_000001 for log-aggregation 2016-11-02 18:18:43,159 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0118 2016-11-02 18:18:44,164 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0118_01_000001] 2016-11-02 18:18:44,164 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0118 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:44,172 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0118 2016-11-02 18:18:44,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0118 2016-11-02 18:18:44,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0118 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:44,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0118 2016-11-02 18:18:44,179 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0123_000001 (auth:SIMPLE) 2016-11-02 18:18:44,184 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0123_01_000001 by user hdfs 2016-11-02 18:18:44,185 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0123 2016-11-02 18:18:44,186 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0123 transitioned from NEW to INITING 2016-11-02 18:18:44,186 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0123_01_000001 to application application_1478007721435_0123 2016-11-02 18:18:44,186 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0123 CONTAINERID=container_e06_1478007721435_0123_01_000001 2016-11-02 18:18:44,187 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:44,198 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0118_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:44,198 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0118/container_e06_1478007721435_0118_01_000001/stderr 2016-11-02 18:18:44,201 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0118/container_e06_1478007721435_0118_01_000001/stdout 2016-11-02 18:18:44,221 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0118 2016-11-02 18:18:44,238 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0123 transitioned from INITING to RUNNING 2016-11-02 18:18:44,238 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0123_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:44,238 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0123 2016-11-02 18:18:44,238 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0123_01_000001 2016-11-02 18:18:44,239 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0123_01_000001.tokens. Credentials list: 2016-11-02 18:18:44,240 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:44,241 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0123_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0123/container_e06_1478007721435_0123_01_000001.tokens 2016-11-02 18:18:44,241 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0123 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0123 2016-11-02 18:18:44,974 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0119_01_000001 is : 15 2016-11-02 18:18:44,974 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0119_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0119_01_000001 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:44,975 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:44,976 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:44,978 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0119_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:44,978 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0119_01_000001 2016-11-02 18:18:45,016 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0119/container_e06_1478007721435_0119_01_000001 2016-11-02 18:18:45,017 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0119 CONTAINERID=container_e06_1478007721435_0119_01_000001 2016-11-02 18:18:45,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0119_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:45,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0119_01_000001 from application application_1478007721435_0119 2016-11-02 18:18:45,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0119_01_000001 2016-11-02 18:18:45,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0119_01_000001 for log-aggregation 2016-11-02 18:18:45,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0119 2016-11-02 18:18:45,685 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0123_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:45,716 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0123_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:45,716 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0123_01_000001 2016-11-02 18:18:45,720 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0123/container_e06_1478007721435_0123_01_000001/default_container_executor.sh] 2016-11-02 18:18:46,023 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0119_01_000001] 2016-11-02 18:18:46,023 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0119 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:46,024 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0119 2016-11-02 18:18:46,024 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0119 2016-11-02 18:18:46,024 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0119 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:46,024 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0119 2016-11-02 18:18:46,038 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0124_000001 (auth:SIMPLE) 2016-11-02 18:18:46,044 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0119_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:46,045 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0119/container_e06_1478007721435_0119_01_000001/stderr 2016-11-02 18:18:46,046 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0119/container_e06_1478007721435_0119_01_000001/stdout 2016-11-02 18:18:46,046 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0124_01_000001 by user hdfs 2016-11-02 18:18:46,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0124 2016-11-02 18:18:46,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0124 transitioned from NEW to INITING 2016-11-02 18:18:46,049 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0124_01_000001 to application application_1478007721435_0124 2016-11-02 18:18:46,052 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0124 CONTAINERID=container_e06_1478007721435_0124_01_000001 2016-11-02 18:18:46,073 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0119 2016-11-02 18:18:46,106 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0124 transitioned from INITING to RUNNING 2016-11-02 18:18:46,107 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0124_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:46,107 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0124 2016-11-02 18:18:46,111 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0124_01_000001 2016-11-02 18:18:46,113 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0124_01_000001.tokens. Credentials list: 2016-11-02 18:18:46,114 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:46,123 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0124_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0124/container_e06_1478007721435_0124_01_000001.tokens 2016-11-02 18:18:46,123 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0124 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0124 2016-11-02 18:18:46,129 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0123_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:47,496 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0124_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:47,530 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0124_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:47,530 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0124_01_000001 2016-11-02 18:18:47,535 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0124/container_e06_1478007721435_0124_01_000001/default_container_executor.sh] 2016-11-02 18:18:49,692 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0124_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:50,356 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0120_01_000001 is : 15 2016-11-02 18:18:50,356 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0120_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0120_01_000001 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:50,356 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:50,356 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:50,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0120_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:50,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0120_01_000001 2016-11-02 18:18:50,395 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0120/container_e06_1478007721435_0120_01_000001 2016-11-02 18:18:50,395 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0120 CONTAINERID=container_e06_1478007721435_0120_01_000001 2016-11-02 18:18:50,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0120_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:50,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0120_01_000001 from application application_1478007721435_0120 2016-11-02 18:18:50,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0120_01_000001 2016-11-02 18:18:50,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0120_01_000001 for log-aggregation 2016-11-02 18:18:50,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0120 2016-11-02 18:18:50,838 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0121_01_000001 is : 15 2016-11-02 18:18:50,838 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0121_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0121_01_000001 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:50,838 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:50,838 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:50,840 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0121_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:50,840 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0121_01_000001 2016-11-02 18:18:50,880 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0121/container_e06_1478007721435_0121_01_000001 2016-11-02 18:18:50,880 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0121 CONTAINERID=container_e06_1478007721435_0121_01_000001 2016-11-02 18:18:50,881 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0121_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:50,881 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0121_01_000001 from application application_1478007721435_0121 2016-11-02 18:18:50,881 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0121_01_000001 2016-11-02 18:18:50,882 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0121_01_000001 for log-aggregation 2016-11-02 18:18:50,882 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0121 2016-11-02 18:18:50,883 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0120_01_000001] 2016-11-02 18:18:50,883 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0120 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:50,883 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0120 2016-11-02 18:18:50,883 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0120 2016-11-02 18:18:50,884 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0120 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:50,884 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0120 2016-11-02 18:18:50,898 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0125_000001 (auth:SIMPLE) 2016-11-02 18:18:50,904 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0125_01_000001 by user hdfs 2016-11-02 18:18:50,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0125 2016-11-02 18:18:50,906 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0125 transitioned from NEW to INITING 2016-11-02 18:18:50,907 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0125 CONTAINERID=container_e06_1478007721435_0125_01_000001 2016-11-02 18:18:50,907 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0125_01_000001 to application application_1478007721435_0125 2016-11-02 18:18:50,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0120_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:50,913 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:50,914 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0120/container_e06_1478007721435_0120_01_000001/stdout 2016-11-02 18:18:50,914 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0120/container_e06_1478007721435_0120_01_000001/stderr 2016-11-02 18:18:50,939 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0120 2016-11-02 18:18:50,962 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0125 transitioned from INITING to RUNNING 2016-11-02 18:18:50,962 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0125_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:50,962 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0125 2016-11-02 18:18:50,963 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0125_01_000001 2016-11-02 18:18:50,965 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0125_01_000001.tokens. Credentials list: 2016-11-02 18:18:50,965 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:50,966 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0125_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0125/container_e06_1478007721435_0125_01_000001.tokens 2016-11-02 18:18:50,967 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0125 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0125 2016-11-02 18:18:51,885 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0121_01_000001] 2016-11-02 18:18:51,885 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0121 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:51,885 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0121 2016-11-02 18:18:51,885 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0121 2016-11-02 18:18:51,886 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0121 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:51,886 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0121 2016-11-02 18:18:51,898 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0126_000001 (auth:SIMPLE) 2016-11-02 18:18:51,911 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0121_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:51,911 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0126_01_000001 by user hdfs 2016-11-02 18:18:51,911 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0121/container_e06_1478007721435_0121_01_000001/stderr 2016-11-02 18:18:51,912 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0121/container_e06_1478007721435_0121_01_000001/stdout 2016-11-02 18:18:51,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0126 2016-11-02 18:18:51,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0126 transitioned from NEW to INITING 2016-11-02 18:18:51,914 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0126_01_000001 to application application_1478007721435_0126 2016-11-02 18:18:51,914 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0126 CONTAINERID=container_e06_1478007721435_0126_01_000001 2016-11-02 18:18:51,930 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0121 2016-11-02 18:18:51,952 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0126 transitioned from INITING to RUNNING 2016-11-02 18:18:51,952 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0126_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:51,952 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0126 2016-11-02 18:18:51,953 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0126_01_000001 2016-11-02 18:18:51,954 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0126_01_000001.tokens. Credentials list: 2016-11-02 18:18:51,954 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:51,955 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0126_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0126/container_e06_1478007721435_0126_01_000001.tokens 2016-11-02 18:18:51,955 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0126 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0126 2016-11-02 18:18:52,058 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0125_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:52,081 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0125_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:52,081 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0125_01_000001 2016-11-02 18:18:52,084 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0125/container_e06_1478007721435_0125_01_000001/default_container_executor.sh] 2016-11-02 18:18:52,846 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0125_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:53,294 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0126_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:53,315 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0126_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:53,315 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0126_01_000001 2016-11-02 18:18:53,318 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0126/container_e06_1478007721435_0126_01_000001/default_container_executor.sh] 2016-11-02 18:18:55,949 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0126_01_000001 since CPU usage is not yet available. 2016-11-02 18:18:56,296 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0122_01_000001 is : 15 2016-11-02 18:18:56,296 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0122_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:56,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:18:56,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0122_01_000001 2016-11-02 18:18:56,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:18:56,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:18:56,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:18:56,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:18:56,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:18:56,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:18:56,297 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:18:56,297 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:18:56,297 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:18:56,297 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:18:56,297 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:18:56,297 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:18:56,297 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:18:56,297 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:18:56,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0122_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:18:56,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0122_01_000001 2016-11-02 18:18:56,318 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0122/container_e06_1478007721435_0122_01_000001 2016-11-02 18:18:56,318 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0122 CONTAINERID=container_e06_1478007721435_0122_01_000001 2016-11-02 18:18:56,319 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0122_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:18:56,319 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0122_01_000001 from application application_1478007721435_0122 2016-11-02 18:18:56,319 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0122_01_000001 2016-11-02 18:18:56,319 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0122_01_000001 for log-aggregation 2016-11-02 18:18:56,319 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0122 2016-11-02 18:18:57,324 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0122_01_000001] 2016-11-02 18:18:57,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0122 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:18:57,325 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0122 2016-11-02 18:18:57,325 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0122 2016-11-02 18:18:57,325 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0122 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:18:57,325 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0122 2016-11-02 18:18:57,345 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0127_000001 (auth:SIMPLE) 2016-11-02 18:18:57,349 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0122_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:18:57,350 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0122/container_e06_1478007721435_0122_01_000001/stderr 2016-11-02 18:18:57,351 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0127_01_000001 by user hdfs 2016-11-02 18:18:57,351 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0122/container_e06_1478007721435_0122_01_000001/stdout 2016-11-02 18:18:57,352 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0127 2016-11-02 18:18:57,352 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0127 transitioned from NEW to INITING 2016-11-02 18:18:57,352 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0127_01_000001 to application application_1478007721435_0127 2016-11-02 18:18:57,353 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0127 CONTAINERID=container_e06_1478007721435_0127_01_000001 2016-11-02 18:18:57,361 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:18:57,375 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0122 2016-11-02 18:18:57,404 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0127 transitioned from INITING to RUNNING 2016-11-02 18:18:57,404 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0127_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:18:57,404 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0127 2016-11-02 18:18:57,404 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0127_01_000001 2016-11-02 18:18:57,410 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0127_01_000001.tokens. Credentials list: 2016-11-02 18:18:57,411 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:18:57,412 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0127_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0127/container_e06_1478007721435_0127_01_000001.tokens 2016-11-02 18:18:57,412 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0127 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0127 2016-11-02 18:18:58,296 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0127_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:18:58,318 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0127_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:18:58,318 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0127_01_000001 2016-11-02 18:18:58,322 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0127/container_e06_1478007721435_0127_01_000001/default_container_executor.sh] 2016-11-02 18:18:59,215 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0127_01_000001 since CPU usage is not yet available. 2016-11-02 18:19:01,501 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0123_01_000001 is : 15 2016-11-02 18:19:01,501 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0123_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0123_01_000001 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:19:01,501 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:01,502 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:19:01,503 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0123_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:19:01,503 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0123_01_000001 2016-11-02 18:19:01,530 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0123/container_e06_1478007721435_0123_01_000001 2016-11-02 18:19:01,530 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0123 CONTAINERID=container_e06_1478007721435_0123_01_000001 2016-11-02 18:19:01,531 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0123_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:19:01,531 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0123_01_000001 from application application_1478007721435_0123 2016-11-02 18:19:01,531 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0123_01_000001 2016-11-02 18:19:01,531 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0123_01_000001 for log-aggregation 2016-11-02 18:19:01,531 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0123 2016-11-02 18:19:02,537 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0123_01_000001] 2016-11-02 18:19:02,537 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0123 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:19:02,537 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0123 2016-11-02 18:19:02,537 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0123 2016-11-02 18:19:02,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0123 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:19:02,538 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0123 2016-11-02 18:19:02,549 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0128_000001 (auth:SIMPLE) 2016-11-02 18:19:02,554 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0128_01_000001 by user hdfs 2016-11-02 18:19:02,555 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0128 2016-11-02 18:19:02,555 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0128 transitioned from NEW to INITING 2016-11-02 18:19:02,555 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0128_01_000001 to application application_1478007721435_0128 2016-11-02 18:19:02,556 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0128 CONTAINERID=container_e06_1478007721435_0128_01_000001 2016-11-02 18:19:02,563 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0123_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:19:02,565 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0123/container_e06_1478007721435_0123_01_000001/stdout 2016-11-02 18:19:02,566 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0123/container_e06_1478007721435_0123_01_000001/stderr 2016-11-02 18:19:02,584 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0123 2016-11-02 18:19:02,595 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0128 transitioned from INITING to RUNNING 2016-11-02 18:19:02,598 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0128_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:19:02,598 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0128 2016-11-02 18:19:02,599 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0128_01_000001 2016-11-02 18:19:02,600 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0128_01_000001.tokens. Credentials list: 2016-11-02 18:19:02,600 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:19:02,603 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0128_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0128/container_e06_1478007721435_0128_01_000001.tokens 2016-11-02 18:19:02,604 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0128 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0128 2016-11-02 18:19:03,279 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0124_01_000001 is : 15 2016-11-02 18:19:03,279 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0124_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0124_01_000001 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:19:03,280 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:03,280 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:19:03,282 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0124_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:19:03,282 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0124_01_000001 2016-11-02 18:19:03,314 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0124/container_e06_1478007721435_0124_01_000001 2016-11-02 18:19:03,314 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0124 CONTAINERID=container_e06_1478007721435_0124_01_000001 2016-11-02 18:19:03,316 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0124_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:19:03,316 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0124_01_000001 from application application_1478007721435_0124 2016-11-02 18:19:03,316 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0124_01_000001 2016-11-02 18:19:03,316 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0124_01_000001 for log-aggregation 2016-11-02 18:19:03,316 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0124 2016-11-02 18:19:03,580 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0128_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:03,603 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0128_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:03,603 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0128_01_000001 2016-11-02 18:19:03,606 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0128/container_e06_1478007721435_0128_01_000001/default_container_executor.sh] 2016-11-02 18:19:04,320 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0124_01_000001] 2016-11-02 18:19:04,321 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0124 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:19:04,321 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0124 2016-11-02 18:19:04,321 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0124 2016-11-02 18:19:04,322 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0124 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:19:04,322 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0124 2016-11-02 18:19:04,334 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0129_000001 (auth:SIMPLE) 2016-11-02 18:19:04,340 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0129_01_000001 by user hdfs 2016-11-02 18:19:04,342 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0124_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:19:04,343 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0124/container_e06_1478007721435_0124_01_000001/stderr 2016-11-02 18:19:04,343 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0129 2016-11-02 18:19:04,347 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0124/container_e06_1478007721435_0124_01_000001/stdout 2016-11-02 18:19:04,347 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0129 transitioned from NEW to INITING 2016-11-02 18:19:04,347 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0129_01_000001 to application application_1478007721435_0129 2016-11-02 18:19:04,347 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0129 CONTAINERID=container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:04,348 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:19:04,372 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0124 2016-11-02 18:19:04,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0129 transitioned from INITING to RUNNING 2016-11-02 18:19:04,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:19:04,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0129 2016-11-02 18:19:04,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:04,399 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0129_01_000001.tokens. Credentials list: 2016-11-02 18:19:04,399 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:19:04,400 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0129_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129/container_e06_1478007721435_0129_01_000001.tokens 2016-11-02 18:19:04,400 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129 2016-11-02 18:19:05,504 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:05,531 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:05,531 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:05,535 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129/container_e06_1478007721435_0129_01_000001/default_container_executor.sh] 2016-11-02 18:19:05,737 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0128_01_000001 since CPU usage is not yet available. 2016-11-02 18:19:05,807 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0129_01_000001 since CPU usage is not yet available. 2016-11-02 18:19:10,296 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0125_01_000001 is : 15 2016-11-02 18:19:10,296 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0125_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0125_01_000001 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:19:10,296 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:10,297 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:19:10,302 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0125_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:19:10,302 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0125_01_000001 2016-11-02 18:19:10,328 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0125/container_e06_1478007721435_0125_01_000001 2016-11-02 18:19:10,329 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0125 CONTAINERID=container_e06_1478007721435_0125_01_000001 2016-11-02 18:19:10,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0125_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:19:10,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0125_01_000001 from application application_1478007721435_0125 2016-11-02 18:19:10,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0125_01_000001 2016-11-02 18:19:10,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0125_01_000001 for log-aggregation 2016-11-02 18:19:10,329 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0125 2016-11-02 18:19:11,332 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0125_01_000001] 2016-11-02 18:19:11,332 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0125 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:19:11,333 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0125 2016-11-02 18:19:11,333 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0125 2016-11-02 18:19:11,333 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0125 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:19:11,333 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0125 2016-11-02 18:19:11,348 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0130_000001 (auth:SIMPLE) 2016-11-02 18:19:11,350 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0125_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:19:11,351 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0125/container_e06_1478007721435_0125_01_000001/stdout 2016-11-02 18:19:11,351 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0125/container_e06_1478007721435_0125_01_000001/stderr 2016-11-02 18:19:11,353 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0130_01_000001 by user hdfs 2016-11-02 18:19:11,355 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0130 2016-11-02 18:19:11,355 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0130 transitioned from NEW to INITING 2016-11-02 18:19:11,355 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0130_01_000001 to application application_1478007721435_0130 2016-11-02 18:19:11,358 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0130 CONTAINERID=container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:11,376 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0125 2016-11-02 18:19:11,394 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0130 transitioned from INITING to RUNNING 2016-11-02 18:19:11,394 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:19:11,394 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0130 2016-11-02 18:19:11,395 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:11,396 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0130_01_000001.tokens. Credentials list: 2016-11-02 18:19:11,396 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:19:11,397 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0130_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130/container_e06_1478007721435_0130_01_000001.tokens 2016-11-02 18:19:11,397 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130 2016-11-02 18:19:11,992 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0126_01_000001 is : 15 2016-11-02 18:19:11,992 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0126_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0126_01_000001 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:19:11,995 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:11,995 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:19:11,997 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0126_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:19:11,997 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0126_01_000001 2016-11-02 18:19:12,023 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0126/container_e06_1478007721435_0126_01_000001 2016-11-02 18:19:12,023 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0126 CONTAINERID=container_e06_1478007721435_0126_01_000001 2016-11-02 18:19:12,023 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0126_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:19:12,023 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0126_01_000001 from application application_1478007721435_0126 2016-11-02 18:19:12,023 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0126_01_000001 2016-11-02 18:19:12,023 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0126_01_000001 for log-aggregation 2016-11-02 18:19:12,023 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0126 2016-11-02 18:19:12,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:12,288 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:12,288 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:12,291 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130/container_e06_1478007721435_0130_01_000001/default_container_executor.sh] 2016-11-02 18:19:13,026 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0126_01_000001] 2016-11-02 18:19:13,027 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0126 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:19:13,027 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0126 2016-11-02 18:19:13,027 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0126 2016-11-02 18:19:13,027 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0126 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:19:13,027 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0126 2016-11-02 18:19:13,039 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0131_000001 (auth:SIMPLE) 2016-11-02 18:19:13,042 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0131_01_000001 by user hdfs 2016-11-02 18:19:13,043 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0131 2016-11-02 18:19:13,043 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0131 transitioned from NEW to INITING 2016-11-02 18:19:13,043 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0131_01_000001 to application application_1478007721435_0131 2016-11-02 18:19:13,044 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0131 CONTAINERID=container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:13,045 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:19:13,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0126_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:19:13,049 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0126/container_e06_1478007721435_0126_01_000001/stdout 2016-11-02 18:19:13,049 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0126/container_e06_1478007721435_0126_01_000001/stderr 2016-11-02 18:19:13,067 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0126 2016-11-02 18:19:13,076 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0131 transitioned from INITING to RUNNING 2016-11-02 18:19:13,076 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:19:13,076 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0131 2016-11-02 18:19:13,076 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:13,077 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0131_01_000001.tokens. Credentials list: 2016-11-02 18:19:13,078 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:19:13,079 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0131_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131/container_e06_1478007721435_0131_01_000001.tokens 2016-11-02 18:19:13,079 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131 2016-11-02 18:19:13,888 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0127_01_000001 is : 15 2016-11-02 18:19:13,888 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0127_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0127_01_000001 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:19:13,889 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:13,889 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:19:13,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0127_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:19:13,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0127_01_000001 2016-11-02 18:19:13,923 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0127/container_e06_1478007721435_0127_01_000001 2016-11-02 18:19:13,924 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0127 CONTAINERID=container_e06_1478007721435_0127_01_000001 2016-11-02 18:19:13,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0127_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:19:13,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0127_01_000001 from application application_1478007721435_0127 2016-11-02 18:19:13,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0127_01_000001 2016-11-02 18:19:13,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0127_01_000001 for log-aggregation 2016-11-02 18:19:13,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0127 2016-11-02 18:19:14,063 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:14,082 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:14,082 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:14,085 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131/container_e06_1478007721435_0131_01_000001/default_container_executor.sh] 2016-11-02 18:19:14,500 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0129_000001 (auth:SIMPLE) 2016-11-02 18:19:14,504 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0129_01_000002 by user hdfs 2016-11-02 18:19:14,505 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0129_01_000002 to application application_1478007721435_0129 2016-11-02 18:19:14,505 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:19:14,505 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.194 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0129 CONTAINERID=container_e06_1478007721435_0129_01_000002 2016-11-02 18:19:14,505 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0129 2016-11-02 18:19:14,509 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000002 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:14,528 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000002 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:14,528 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0129_01_000002 2016-11-02 18:19:14,530 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129/container_e06_1478007721435_0129_01_000002/default_container_executor.sh] 2016-11-02 18:19:14,928 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0127_01_000001] 2016-11-02 18:19:14,928 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0127 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:19:14,929 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0127 2016-11-02 18:19:14,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0127 2016-11-02 18:19:14,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0127 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:19:14,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0127 2016-11-02 18:19:14,940 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0132_000001 (auth:SIMPLE) 2016-11-02 18:19:14,944 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0132_01_000001 by user hdfs 2016-11-02 18:19:14,945 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0132 2016-11-02 18:19:14,945 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0132 transitioned from NEW to INITING 2016-11-02 18:19:14,945 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0132_01_000001 to application application_1478007721435_0132 2016-11-02 18:19:14,946 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0132 CONTAINERID=container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:14,949 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0127_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:19:14,950 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0127/container_e06_1478007721435_0127_01_000001/stderr 2016-11-02 18:19:14,956 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0127/container_e06_1478007721435_0127_01_000001/stdout 2016-11-02 18:19:14,975 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0127 2016-11-02 18:19:14,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0132 transitioned from INITING to RUNNING 2016-11-02 18:19:14,990 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:19:14,990 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0132 2016-11-02 18:19:14,990 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:14,992 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0132_01_000001.tokens. Credentials list: 2016-11-02 18:19:14,992 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:19:14,996 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0132_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132/container_e06_1478007721435_0132_01_000001.tokens 2016-11-02 18:19:14,997 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132 2016-11-02 18:19:15,135 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0130_01_000001 since CPU usage is not yet available. 2016-11-02 18:19:15,159 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0131_01_000001 since CPU usage is not yet available. 2016-11-02 18:19:15,212 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0129_01_000002 since CPU usage is not yet available. 2016-11-02 18:19:16,607 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:16,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:16,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:16,635 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132/container_e06_1478007721435_0132_01_000001/default_container_executor.sh] 2016-11-02 18:19:17,074 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0128_01_000001 is : 15 2016-11-02 18:19:17,074 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e06_1478007721435_0128_01_000001 and exit code: 15 ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) at org.apache.hadoop.util.Shell.run(Shell.java:850) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e06_1478007721435_0128_01_000001 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 15 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=15: 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:946) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:850) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2016-11-02 18:19:17,075 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:19:17,078 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 15. 2016-11-02 18:19:17,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0128_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2016-11-02 18:19:17,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0128_01_000001 2016-11-02 18:19:17,113 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0128/container_e06_1478007721435_0128_01_000001 2016-11-02 18:19:17,115 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1478007721435_0128 CONTAINERID=container_e06_1478007721435_0128_01_000001 2016-11-02 18:19:17,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0128_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2016-11-02 18:19:17,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0128_01_000001 from application application_1478007721435_0128 2016-11-02 18:19:17,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0128_01_000001 2016-11-02 18:19:17,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0128_01_000001 for log-aggregation 2016-11-02 18:19:17,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0128 2016-11-02 18:19:18,120 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0128_01_000001] 2016-11-02 18:19:18,121 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0128 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:19:18,122 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0128 2016-11-02 18:19:18,122 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0128 2016-11-02 18:19:18,122 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0128 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:19:18,122 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0128 2016-11-02 18:19:18,132 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0133_000001 (auth:SIMPLE) 2016-11-02 18:19:18,141 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0133_01_000001 by user hdfs 2016-11-02 18:19:18,143 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0133 2016-11-02 18:19:18,143 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0133 transitioned from NEW to INITING 2016-11-02 18:19:18,143 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0133_01_000001 to application application_1478007721435_0133 2016-11-02 18:19:18,144 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0133 CONTAINERID=container_e06_1478007721435_0133_01_000001 2016-11-02 18:19:18,146 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:19:18,156 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0128_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:19:18,158 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0128/container_e06_1478007721435_0128_01_000001/stdout 2016-11-02 18:19:18,158 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0128/container_e06_1478007721435_0128_01_000001/stderr 2016-11-02 18:19:18,218 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0128 2016-11-02 18:19:18,225 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0133 transitioned from INITING to RUNNING 2016-11-02 18:19:18,225 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0133_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:19:18,225 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0133 2016-11-02 18:19:18,226 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0133_01_000001 2016-11-02 18:19:18,235 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0133_01_000001.tokens. Credentials list: 2016-11-02 18:19:18,235 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs 2016-11-02 18:19:18,237 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-yarn/nm-local-dir/nmPrivate/container_e06_1478007721435_0133_01_000001.tokens to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0133/container_e06_1478007721435_0133_01_000001.tokens 2016-11-02 18:19:18,238 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0133 = file:/tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0133 2016-11-02 18:19:18,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0132_01_000001 since CPU usage is not yet available. 2016-11-02 18:19:19,007 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0129_000001 (auth:SIMPLE) 2016-11-02 18:19:19,026 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0129_01_000003 by user hdfs 2016-11-02 18:19:19,029 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.194 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0129 CONTAINERID=container_e06_1478007721435_0129_01_000003 2016-11-02 18:19:19,029 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0129_01_000003 to application application_1478007721435_0129 2016-11-02 18:19:19,029 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:19:19,029 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0129 2016-11-02 18:19:19,029 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000003 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:19,057 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000003 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:19,057 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0129_01_000003 2016-11-02 18:19:19,063 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129/container_e06_1478007721435_0129_01_000003/default_container_executor.sh] 2016-11-02 18:19:20,429 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0133_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:20,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0133_01_000001 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:20,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0133_01_000001 2016-11-02 18:19:20,470 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0133/container_e06_1478007721435_0133_01_000001/default_container_executor.sh] 2016-11-02 18:19:21,660 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0129_01_000003 since CPU usage is not yet available. 2016-11-02 18:19:21,952 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0133_01_000001 since CPU usage is not yet available. 2016-11-02 18:19:22,193 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0130_000001 (auth:SIMPLE) 2016-11-02 18:19:22,200 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0130_01_000002 by user hdfs 2016-11-02 18:19:22,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0130_01_000002 to application application_1478007721435_0130 2016-11-02 18:19:22,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:19:22,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0130 2016-11-02 18:19:22,202 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.194 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0130 CONTAINERID=container_e06_1478007721435_0130_01_000002 2016-11-02 18:19:22,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000002 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:22,227 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000002 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:22,227 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0130_01_000002 2016-11-02 18:19:22,240 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130/container_e06_1478007721435_0130_01_000002/default_container_executor.sh] 2016-11-02 18:19:23,762 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0130_000001 (auth:SIMPLE) 2016-11-02 18:19:23,773 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0130_01_000003 by user hdfs 2016-11-02 18:19:23,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0130_01_000003 to application application_1478007721435_0130 2016-11-02 18:19:23,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:19:23,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0130 2016-11-02 18:19:23,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000003 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:23,777 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.194 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0130 CONTAINERID=container_e06_1478007721435_0130_01_000003 2016-11-02 18:19:23,811 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000003 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:23,811 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0130_01_000003 2016-11-02 18:19:23,814 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130/container_e06_1478007721435_0130_01_000003/default_container_executor.sh] 2016-11-02 18:19:25,152 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0130_01_000002 since CPU usage is not yet available. 2016-11-02 18:19:25,259 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0130_01_000003 since CPU usage is not yet available. 2016-11-02 18:19:29,455 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0131_000001 (auth:SIMPLE) 2016-11-02 18:19:29,461 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0131_01_000002 by user hdfs 2016-11-02 18:19:29,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0131_01_000002 to application application_1478007721435_0131 2016-11-02 18:19:29,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:19:29,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0131 2016-11-02 18:19:29,463 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.194 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0131 CONTAINERID=container_e06_1478007721435_0131_01_000002 2016-11-02 18:19:29,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000002 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:29,488 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000002 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:29,488 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0131_01_000002 2016-11-02 18:19:29,490 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131/container_e06_1478007721435_0131_01_000002/default_container_executor.sh] 2016-11-02 18:19:30,188 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0130_01_000004 2016-11-02 18:19:30,537 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0131_000001 (auth:SIMPLE) 2016-11-02 18:19:30,543 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0131_01_000003 by user hdfs 2016-11-02 18:19:30,545 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0131_01_000003 to application application_1478007721435_0131 2016-11-02 18:19:30,545 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:19:30,545 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0131 2016-11-02 18:19:30,545 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.194 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0131 CONTAINERID=container_e06_1478007721435_0131_01_000003 2016-11-02 18:19:30,546 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000003 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:30,575 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000003 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:30,575 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0131_01_000003 2016-11-02 18:19:30,580 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131/container_e06_1478007721435_0131_01_000003/default_container_executor.sh] 2016-11-02 18:19:32,076 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0131_01_000003 since CPU usage is not yet available. 2016-11-02 18:19:32,132 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_e06_1478007721435_0131_01_000002 since CPU usage is not yet available. 2016-11-02 18:19:36,818 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0132_000001 (auth:SIMPLE) 2016-11-02 18:19:36,818 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0132_000001 (auth:SIMPLE) 2016-11-02 18:19:36,825 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0132_01_000003 by user hdfs 2016-11-02 18:19:36,827 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0132_01_000003 to application application_1478007721435_0132 2016-11-02 18:19:36,828 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:19:36,828 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0132 2016-11-02 18:19:36,828 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000003 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:36,828 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.194 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0132 CONTAINERID=container_e06_1478007721435_0132_01_000003 2016-11-02 18:19:36,829 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0132_01_000002 by user hdfs 2016-11-02 18:19:36,831 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.194 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0132 CONTAINERID=container_e06_1478007721435_0132_01_000002 2016-11-02 18:19:36,832 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0132_01_000002 to application application_1478007721435_0132 2016-11-02 18:19:36,832 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:19:36,832 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0132 2016-11-02 18:19:36,832 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000002 transitioned from LOCALIZING to LOCALIZED 2016-11-02 18:19:36,862 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000002 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:36,863 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0132_01_000002 2016-11-02 18:19:36,871 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132/container_e06_1478007721435_0132_01_000002/default_container_executor.sh] 2016-11-02 18:19:36,878 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000003 transitioned from LOCALIZED to RUNNING 2016-11-02 18:19:36,878 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e06_1478007721435_0132_01_000003 2016-11-02 18:19:36,886 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132/container_e06_1478007721435_0132_01_000003/default_container_executor.sh] 2016-11-02 18:19:37,193 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /tmp/hadoop-yarn/nm-local-dir error, used space above threshold of 40.0%, removing from list of valid directories 2016-11-02 18:19:37,193 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed: 1/1 local-dirs are bad: /tmp/hadoop-yarn/nm-local-dir; 2016-11-02 18:19:37,193 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs are bad: /tmp/hadoop-yarn/nm-local-dir; 2016-11-02 18:19:37,237 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0130_000001 (auth:SIMPLE) 2016-11-02 18:19:37,241 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0132_000001 (auth:SIMPLE) 2016-11-02 18:19:37,247 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0131_000001 (auth:SIMPLE) 2016-11-02 18:19:37,247 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0129_000001 (auth:SIMPLE) 2016-11-02 18:19:37,265 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:37,265 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:37,266 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0129 CONTAINERID=container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:37,266 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0131 CONTAINERID=container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:37,266 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000001 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,266 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:37,268 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0130 CONTAINERID=container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:37,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:37,266 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000001 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,268 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0132 CONTAINERID=container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:37,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:37,299 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0129_01_000001 is : 143 2016-11-02 18:19:37,323 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:37,363 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0131_01_000001 is : 143 2016-11-02 18:19:37,387 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000001 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,387 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000001 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,390 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000001 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,390 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000001 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,390 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:37,421 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0130_01_000001 is : 143 2016-11-02 18:19:37,449 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:37,478 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0132_01_000001 is : 143 2016-11-02 18:19:37,523 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129/container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:37,531 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131/container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:37,533 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000001 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,533 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000001 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,534 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0129 CONTAINERID=container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:37,549 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000001 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,550 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0131 CONTAINERID=container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:37,560 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000001 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,560 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130/container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:37,561 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0131_01_000001, container_e06_1478007721435_0129_01_000001] 2016-11-02 18:19:37,573 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132/container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:37,573 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0129_01_000001 from application application_1478007721435_0129 2016-11-02 18:19:37,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:37,574 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Log aggregation cannot be started for container_e06_1478007721435_0129_01_000001, as its an absent container 2016-11-02 18:19:37,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0129 2016-11-02 18:19:37,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0131_01_000001 from application application_1478007721435_0131 2016-11-02 18:19:37,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:37,574 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Log aggregation cannot be started for container_e06_1478007721435_0131_01_000001, as its an absent container 2016-11-02 18:19:37,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0131 2016-11-02 18:19:37,574 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0130 CONTAINERID=container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:37,576 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000001 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,577 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0129_01_000001 2016-11-02 18:19:37,577 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000002 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,577 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000003 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,577 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000002 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,577 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000003 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,577 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0131_01_000001 2016-11-02 18:19:37,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000002 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000003 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000002 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000003 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0133_01_000001 transitioned from RUNNING to KILLING 2016-11-02 18:19:37,578 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0130_01_000001] 2016-11-02 18:19:37,578 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0132 CONTAINERID=container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:37,581 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000001 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,581 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0130_01_000001 from application application_1478007721435_0130 2016-11-02 18:19:37,581 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0130_01_000001 2016-11-02 18:19:37,581 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Log aggregation cannot be started for container_e06_1478007721435_0130_01_000001, as its an absent container 2016-11-02 18:19:37,581 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0130 2016-11-02 18:19:37,581 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0129_01_000002 2016-11-02 18:19:37,588 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0132_01_000001] 2016-11-02 18:19:37,598 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0129_01_000002 is : 143 2016-11-02 18:19:37,627 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0129_01_000003 2016-11-02 18:19:37,648 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0129_01_000003 is : 143 2016-11-02 18:19:37,682 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0130_01_000002 2016-11-02 18:19:37,697 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0130_01_000002 is : 143 2016-11-02 18:19:37,712 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0130_01_000003 2016-11-02 18:19:37,732 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0130_01_000003 is : 143 2016-11-02 18:19:37,758 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0131_01_000002 2016-11-02 18:19:37,776 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0131_01_000002 is : 143 2016-11-02 18:19:37,796 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0131_01_000003 2016-11-02 18:19:37,809 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0131_01_000003 is : 143 2016-11-02 18:19:37,824 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0132_01_000002 2016-11-02 18:19:37,834 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0132_01_000002 is : 143 2016-11-02 18:19:37,856 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0132_01_000003 2016-11-02 18:19:37,865 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0132_01_000003 is : 143 2016-11-02 18:19:37,887 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e06_1478007721435_0133_01_000001 2016-11-02 18:19:37,899 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e06_1478007721435_0133_01_000001 is : 143 2016-11-02 18:19:37,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0132_01_000001 from application application_1478007721435_0132 2016-11-02 18:19:37,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0132_01_000001 2016-11-02 18:19:37,924 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Log aggregation cannot be started for container_e06_1478007721435_0132_01_000001, as its an absent container 2016-11-02 18:19:37,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0132 2016-11-02 18:19:37,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000003 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000003 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,925 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000003 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,925 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,925 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000003 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,925 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0133_01_000001 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-11-02 18:19:37,925 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129/container_e06_1478007721435_0129_01_000002 2016-11-02 18:19:37,926 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0129/container_e06_1478007721435_0129_01_000003 2016-11-02 18:19:37,926 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130/container_e06_1478007721435_0130_01_000002 2016-11-02 18:19:37,926 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0130/container_e06_1478007721435_0130_01_000003 2016-11-02 18:19:37,926 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131/container_e06_1478007721435_0131_01_000002 2016-11-02 18:19:37,927 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0131/container_e06_1478007721435_0131_01_000003 2016-11-02 18:19:37,927 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132/container_e06_1478007721435_0132_01_000002 2016-11-02 18:19:37,927 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132/container_e06_1478007721435_0132_01_000003 2016-11-02 18:19:37,928 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0133/container_e06_1478007721435_0133_01_000001 2016-11-02 18:19:37,929 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0129 CONTAINERID=container_e06_1478007721435_0129_01_000002 2016-11-02 18:19:37,930 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,930 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0129 CONTAINERID=container_e06_1478007721435_0129_01_000003 2016-11-02 18:19:37,930 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0129_01_000003 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,930 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0130 CONTAINERID=container_e06_1478007721435_0130_01_000002 2016-11-02 18:19:37,931 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,931 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0130 CONTAINERID=container_e06_1478007721435_0130_01_000003 2016-11-02 18:19:37,931 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0130_01_000003 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,932 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0131 CONTAINERID=container_e06_1478007721435_0131_01_000002 2016-11-02 18:19:37,932 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,932 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0131 CONTAINERID=container_e06_1478007721435_0131_01_000003 2016-11-02 18:19:37,933 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0131_01_000003 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,933 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0132 CONTAINERID=container_e06_1478007721435_0132_01_000002 2016-11-02 18:19:37,933 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,933 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0132 CONTAINERID=container_e06_1478007721435_0132_01_000003 2016-11-02 18:19:37,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0132_01_000003 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,934 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478007721435_0133 CONTAINERID=container_e06_1478007721435_0133_01_000001 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0133_01_000001 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0129_01_000002 from application application_1478007721435_0129 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0129_01_000002 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0129_01_000002 for log-aggregation 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0129 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0129_01_000003 from application application_1478007721435_0129 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0129_01_000003 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0129_01_000003 for log-aggregation 2016-11-02 18:19:37,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0129 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0130_01_000002 from application application_1478007721435_0130 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0130_01_000002 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0130_01_000002 for log-aggregation 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0130 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0130_01_000003 from application application_1478007721435_0130 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0130_01_000003 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0130_01_000003 for log-aggregation 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0130 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0131_01_000002 from application application_1478007721435_0131 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0131_01_000002 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0131_01_000002 for log-aggregation 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0131 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0131_01_000003 from application application_1478007721435_0131 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0131_01_000003 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0131_01_000003 for log-aggregation 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0131 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0132_01_000002 from application application_1478007721435_0132 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0132_01_000002 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0132_01_000002 for log-aggregation 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0132 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0132_01_000003 from application application_1478007721435_0132 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0132_01_000003 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0133_01_000001] 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0132_01_000003 for log-aggregation 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0132 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0133_01_000001 from application application_1478007721435_0133 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0133_01_000001 2016-11-02 18:19:37,936 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Log aggregation cannot be started for container_e06_1478007721435_0133_01_000001, as its an absent container 2016-11-02 18:19:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0133 2016-11-02 18:21:34,164 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0132 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:21:34,166 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0132 2016-11-02 18:21:34,166 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0132 2016-11-02 18:21:34,175 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0132 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:21:34,175 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0132 2016-11-02 18:21:34,211 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0132_01_000002. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:34,212 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0132_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:34,213 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0132/container_e06_1478007721435_0132_01_000002/stdout 2016-11-02 18:21:34,214 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0132/container_e06_1478007721435_0132_01_000002/stderr 2016-11-02 18:21:34,214 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0132/container_e06_1478007721435_0132_01_000003/stdout 2016-11-02 18:21:34,214 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0132/container_e06_1478007721435_0132_01_000003/stderr 2016-11-02 18:21:34,247 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0132 2016-11-02 18:21:35,165 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0133 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:21:35,171 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache/application_1478007721435_0133 2016-11-02 18:21:35,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0133 2016-11-02 18:21:35,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0133 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:21:35,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0133 2016-11-02 18:21:35,173 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0133 2016-11-02 18:21:37,187 INFO org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /tmp/hadoop-yarn/nm-local-dir passed disk check, adding to list of valid directories. 2016-11-02 18:21:37,193 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Permissions incorrectly set for dir /tmp/hadoop-yarn/nm-local-dir/usercache, should be rw-r-----, actual value = rwxr-xr-x 2016-11-02 18:21:37,193 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Attempting to initialize /tmp/hadoop-yarn/nm-local-dir 2016-11-02 18:21:37,209 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) turned good: 1/1 local-dirs are good: /tmp/hadoop-yarn/nm-local-dir; 1/1 log-dirs are good: /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:39,194 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0144_000001 (auth:SIMPLE) 2016-11-02 18:21:39,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0144_01_000001 by user hdfs 2016-11-02 18:21:39,204 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0144 2016-11-02 18:21:39,205 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0144 transitioned from NEW to INITING 2016-11-02 18:21:39,205 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0144_01_000001 to application application_1478007721435_0144 2016-11-02 18:21:39,205 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0144 CONTAINERID=container_e06_1478007721435_0144_01_000001 2016-11-02 18:21:39,263 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0144 transitioned from INITING to RUNNING 2016-11-02 18:21:39,263 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0144_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:21:39,263 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0144 2016-11-02 18:21:39,263 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0144_01_000001 2016-11-02 18:21:39,264 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:21:39,265 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0144_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:21:39,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0144_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:21:39,267 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:21:39,267 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:21:39,267 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0144 CONTAINERID=container_e06_1478007721435_0144_01_000001 2016-11-02 18:21:39,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0144_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:21:39,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0144_01_000001 from application application_1478007721435_0144 2016-11-02 18:21:39,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0144_01_000001 2016-11-02 18:21:39,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0144_01_000001 for log-aggregation 2016-11-02 18:21:39,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0144 2016-11-02 18:21:40,271 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0144_01_000001] 2016-11-02 18:21:40,273 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0144 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:21:40,277 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0144 2016-11-02 18:21:40,278 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0144 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:21:40,278 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0144 2016-11-02 18:21:40,301 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0144_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:46,299 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0147_000001 (auth:SIMPLE) 2016-11-02 18:21:46,307 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0147_01_000001 by user hdfs 2016-11-02 18:21:46,308 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0147 2016-11-02 18:21:46,308 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0147 transitioned from NEW to INITING 2016-11-02 18:21:46,308 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0147_01_000001 to application application_1478007721435_0147 2016-11-02 18:21:46,309 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0147 CONTAINERID=container_e06_1478007721435_0147_01_000001 2016-11-02 18:21:46,310 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:21:46,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0147 transitioned from INITING to RUNNING 2016-11-02 18:21:46,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0147_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:21:46,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0147 2016-11-02 18:21:46,359 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0147_01_000001 2016-11-02 18:21:46,359 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:21:46,360 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0147_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:21:46,360 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0147_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:21:46,361 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:21:46,361 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:21:46,361 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0147 CONTAINERID=container_e06_1478007721435_0147_01_000001 2016-11-02 18:21:46,362 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0147_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:21:46,362 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0147_01_000001 from application application_1478007721435_0147 2016-11-02 18:21:46,362 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0147_01_000001 2016-11-02 18:21:46,362 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0147_01_000001 for log-aggregation 2016-11-02 18:21:46,362 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0147 2016-11-02 18:21:47,365 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0147_01_000001] 2016-11-02 18:21:47,365 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0147 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:21:47,365 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0147 2016-11-02 18:21:47,366 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0147 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:21:47,366 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0147 2016-11-02 18:21:47,388 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0147_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:51,372 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0130 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:21:51,372 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0131 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:21:51,372 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0130 2016-11-02 18:21:51,372 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0131 2016-11-02 18:21:51,373 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0130 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:21:51,373 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0131 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:21:51,373 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0130 2016-11-02 18:21:51,373 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0131 2016-11-02 18:21:51,412 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0130_01_000002. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:51,415 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0130_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:51,415 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0130/container_e06_1478007721435_0130_01_000002/stdout 2016-11-02 18:21:51,416 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0130/container_e06_1478007721435_0130_01_000002/stderr 2016-11-02 18:21:51,416 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0130/container_e06_1478007721435_0130_01_000003/stdout 2016-11-02 18:21:51,416 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0130/container_e06_1478007721435_0130_01_000003/stderr 2016-11-02 18:21:51,417 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0131_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:51,418 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0131_01_000002. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:51,418 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0131/container_e06_1478007721435_0131_01_000003/stdout 2016-11-02 18:21:51,418 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0131/container_e06_1478007721435_0131_01_000003/stderr 2016-11-02 18:21:51,419 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0131/container_e06_1478007721435_0131_01_000002/stdout 2016-11-02 18:21:51,419 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0131/container_e06_1478007721435_0131_01_000002/stderr 2016-11-02 18:21:51,433 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0130 2016-11-02 18:21:51,437 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0131 2016-11-02 18:21:55,378 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0129 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:21:55,378 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0129 2016-11-02 18:21:55,379 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0129 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:21:55,379 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0129 2016-11-02 18:21:55,410 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0129_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:55,411 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0129/container_e06_1478007721435_0129_01_000003/stderr 2016-11-02 18:21:55,411 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0129/container_e06_1478007721435_0129_01_000003/stdout 2016-11-02 18:21:55,412 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0129_01_000002. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:21:55,412 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0129/container_e06_1478007721435_0129_01_000002/stderr 2016-11-02 18:21:55,412 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0129/container_e06_1478007721435_0129_01_000002/stdout 2016-11-02 18:21:55,428 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/userlogs/application_1478007721435_0129 2016-11-02 18:21:57,728 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0146_000001 (auth:SIMPLE) 2016-11-02 18:21:57,733 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0146_01_000003 by user hdfs 2016-11-02 18:21:57,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0146 2016-11-02 18:21:57,735 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0146 transitioned from NEW to INITING 2016-11-02 18:21:57,735 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0146_01_000003 to application application_1478007721435_0146 2016-11-02 18:21:57,735 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.198 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0146 CONTAINERID=container_e06_1478007721435_0146_01_000003 2016-11-02 18:21:57,766 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0146 transitioned from INITING to RUNNING 2016-11-02 18:21:57,766 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0146_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:21:57,766 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0146 2016-11-02 18:21:57,766 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0146_01_000003 2016-11-02 18:21:57,767 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:21:57,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0146_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:21:57,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0146_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:21:57,768 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:21:57,768 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:21:57,768 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0146 CONTAINERID=container_e06_1478007721435_0146_01_000003 2016-11-02 18:21:57,769 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0146_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:21:57,769 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0146_01_000003 from application application_1478007721435_0146 2016-11-02 18:21:57,769 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0146_01_000003 2016-11-02 18:21:57,769 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0146_01_000003 for log-aggregation 2016-11-02 18:21:57,769 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0146 2016-11-02 18:22:03,779 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0146_01_000003] 2016-11-02 18:22:05,579 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0145_000001 (auth:SIMPLE) 2016-11-02 18:22:05,586 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0145_01_000002 by user hdfs 2016-11-02 18:22:05,590 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0145 2016-11-02 18:22:05,591 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0145 transitioned from NEW to INITING 2016-11-02 18:22:05,591 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0145_01_000002 to application application_1478007721435_0145 2016-11-02 18:22:05,592 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.199 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0145 CONTAINERID=container_e06_1478007721435_0145_01_000002 2016-11-02 18:22:05,593 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:22:05,624 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0145 transitioned from INITING to RUNNING 2016-11-02 18:22:05,624 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0145_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:22:05,624 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0145 2016-11-02 18:22:05,624 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0145_01_000002 2016-11-02 18:22:05,625 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:22:05,625 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0145_01_000002.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:22:05,626 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0145_01_000002 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:22:05,626 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:22:05,626 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:22:05,626 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0145 CONTAINERID=container_e06_1478007721435_0145_01_000002 2016-11-02 18:22:05,627 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0145_01_000002 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:22:05,627 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0145_01_000002 from application application_1478007721435_0145 2016-11-02 18:22:05,627 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0145_01_000002 2016-11-02 18:22:05,627 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0145_01_000002 for log-aggregation 2016-11-02 18:22:05,627 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0145 2016-11-02 18:22:10,211 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0148_000001 (auth:SIMPLE) 2016-11-02 18:22:10,221 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0148_01_000002 by user hdfs 2016-11-02 18:22:10,225 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0148 2016-11-02 18:22:10,226 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0148 transitioned from NEW to INITING 2016-11-02 18:22:10,226 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0148_01_000002 to application application_1478007721435_0148 2016-11-02 18:22:10,226 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.199 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0148 CONTAINERID=container_e06_1478007721435_0148_01_000002 2016-11-02 18:22:10,265 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0148 transitioned from INITING to RUNNING 2016-11-02 18:22:10,265 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0148_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:22:10,265 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0148 2016-11-02 18:22:10,265 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0148_01_000002 2016-11-02 18:22:10,266 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:22:10,266 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0148_01_000002.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:22:10,266 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0148_01_000002 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:22:10,267 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:22:10,267 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:22:10,267 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0148 CONTAINERID=container_e06_1478007721435_0148_01_000002 2016-11-02 18:22:10,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0148_01_000002 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:22:10,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0148_01_000002 from application application_1478007721435_0148 2016-11-02 18:22:10,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0148_01_000002 2016-11-02 18:22:10,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0148_01_000002 for log-aggregation 2016-11-02 18:22:10,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0148 2016-11-02 18:22:12,272 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0145_01_000002] 2016-11-02 18:22:16,248 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0149_000001 (auth:SIMPLE) 2016-11-02 18:22:16,253 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0149_01_000003 by user hdfs 2016-11-02 18:22:16,254 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0149 2016-11-02 18:22:16,254 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0149 transitioned from NEW to INITING 2016-11-02 18:22:16,257 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0149_01_000003 to application application_1478007721435_0149 2016-11-02 18:22:16,258 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.199 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0149 CONTAINERID=container_e06_1478007721435_0149_01_000003 2016-11-02 18:22:16,259 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:22:16,278 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0148_01_000002] 2016-11-02 18:22:16,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0149 transitioned from INITING to RUNNING 2016-11-02 18:22:16,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0149_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:22:16,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0149 2016-11-02 18:22:16,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0149_01_000003 2016-11-02 18:22:16,289 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:22:16,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0149_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:22:16,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0149_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:22:16,290 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:22:16,290 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:22:16,291 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0149 CONTAINERID=container_e06_1478007721435_0149_01_000003 2016-11-02 18:22:16,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0149_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:22:16,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0149_01_000003 from application application_1478007721435_0149 2016-11-02 18:22:16,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0149_01_000003 2016-11-02 18:22:16,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0149_01_000003 for log-aggregation 2016-11-02 18:22:16,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0149 2016-11-02 18:22:18,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0146 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:22:18,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0146 2016-11-02 18:22:18,296 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0146 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:22:18,296 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0146 2016-11-02 18:22:18,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0146_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:22:22,301 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0149_01_000003] 2016-11-02 18:22:22,302 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0149_01_000004 2016-11-02 18:22:23,106 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0149_000001 (auth:SIMPLE) 2016-11-02 18:22:23,114 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0149_01_000005 by user hdfs 2016-11-02 18:22:23,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0149_01_000005 to application application_1478007721435_0149 2016-11-02 18:22:23,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0149_01_000005 transitioned from NEW to LOCALIZING 2016-11-02 18:22:23,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0149 2016-11-02 18:22:23,119 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0149_01_000005 2016-11-02 18:22:23,119 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.199 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0149 CONTAINERID=container_e06_1478007721435_0149_01_000005 2016-11-02 18:22:23,120 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:22:23,120 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0149_01_000005.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:22:23,120 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0149_01_000005 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:22:23,121 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0149 CONTAINERID=container_e06_1478007721435_0149_01_000005 2016-11-02 18:22:23,123 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0149_01_000005 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:22:23,123 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0149_01_000005 from application application_1478007721435_0149 2016-11-02 18:22:23,123 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0149_01_000005 2016-11-02 18:22:23,123 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0149_01_000005 for log-aggregation 2016-11-02 18:22:23,123 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0149 2016-11-02 18:22:23,124 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:22:23,124 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:22:29,133 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0149_01_000005] 2016-11-02 18:22:33,139 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0148 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:22:33,139 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0148 2016-11-02 18:22:33,139 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0148 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:22:33,140 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0148 2016-11-02 18:22:33,161 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0148_01_000002. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:22:37,144 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0145 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:22:37,147 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0145 2016-11-02 18:22:37,148 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0145 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:22:37,148 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0145 2016-11-02 18:22:37,163 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0145_01_000002. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:22:38,146 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0149 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:22:38,146 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0149 2016-11-02 18:22:38,148 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0149 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:22:38,148 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0149 2016-11-02 18:22:38,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0149_01_000005. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:22:38,173 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0149_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:24:01,627 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0150_000001 (auth:SIMPLE) 2016-11-02 18:24:01,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0150_01_000003 by user hdfs 2016-11-02 18:24:01,647 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0150 2016-11-02 18:24:01,648 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0150 transitioned from NEW to INITING 2016-11-02 18:24:01,648 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0150_01_000003 to application application_1478007721435_0150 2016-11-02 18:24:01,648 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.195 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0150 CONTAINERID=container_e06_1478007721435_0150_01_000003 2016-11-02 18:24:01,700 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0150 transitioned from INITING to RUNNING 2016-11-02 18:24:01,700 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:24:01,700 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0150 2016-11-02 18:24:01,700 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0150_01_000003 2016-11-02 18:24:01,701 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:01,701 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0150_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:01,702 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:24:01,702 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:24:01,702 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:24:01,703 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0150 CONTAINERID=container_e06_1478007721435_0150_01_000003 2016-11-02 18:24:01,703 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:24:01,703 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0150_01_000003 from application application_1478007721435_0150 2016-11-02 18:24:01,703 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0150_01_000003 2016-11-02 18:24:01,703 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0150_01_000003 for log-aggregation 2016-11-02 18:24:01,704 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0150 2016-11-02 18:24:07,715 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0150_01_000003] 2016-11-02 18:24:07,715 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0150_01_000004 2016-11-02 18:24:07,850 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0150_000001 (auth:SIMPLE) 2016-11-02 18:24:07,861 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0150_01_000005 by user hdfs 2016-11-02 18:24:07,863 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0150_01_000005 to application application_1478007721435_0150 2016-11-02 18:24:07,863 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000005 transitioned from NEW to LOCALIZING 2016-11-02 18:24:07,863 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0150 2016-11-02 18:24:07,864 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.195 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0150 CONTAINERID=container_e06_1478007721435_0150_01_000005 2016-11-02 18:24:07,864 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0150_01_000005 2016-11-02 18:24:07,864 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:07,865 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0150_01_000005.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:07,865 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000005 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:24:07,866 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:24:07,866 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:24:07,866 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0150 CONTAINERID=container_e06_1478007721435_0150_01_000005 2016-11-02 18:24:07,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000005 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:24:07,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0150_01_000005 from application application_1478007721435_0150 2016-11-02 18:24:07,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0150_01_000005 2016-11-02 18:24:07,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0150_01_000005 for log-aggregation 2016-11-02 18:24:07,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0150 2016-11-02 18:24:13,877 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0150_01_000005] 2016-11-02 18:24:14,061 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0150_000001 (auth:SIMPLE) 2016-11-02 18:24:14,065 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0150_01_000006 by user hdfs 2016-11-02 18:24:14,066 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0150_01_000006 to application application_1478007721435_0150 2016-11-02 18:24:14,066 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000006 transitioned from NEW to LOCALIZING 2016-11-02 18:24:14,066 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0150 2016-11-02 18:24:14,067 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.195 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0150 CONTAINERID=container_e06_1478007721435_0150_01_000006 2016-11-02 18:24:14,067 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0150_01_000006 2016-11-02 18:24:14,067 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:14,067 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0150_01_000006.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:14,069 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000006 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:24:14,070 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:24:14,070 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:24:14,070 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0150 CONTAINERID=container_e06_1478007721435_0150_01_000006 2016-11-02 18:24:14,071 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000006 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:24:14,071 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0150_01_000006 from application application_1478007721435_0150 2016-11-02 18:24:14,071 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0150_01_000006 2016-11-02 18:24:14,071 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0150_01_000006 for log-aggregation 2016-11-02 18:24:14,071 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0150 2016-11-02 18:24:20,081 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0150_01_000006] 2016-11-02 18:24:20,275 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0150_000001 (auth:SIMPLE) 2016-11-02 18:24:20,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0150_01_000007 by user hdfs 2016-11-02 18:24:20,294 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0150_01_000007 to application application_1478007721435_0150 2016-11-02 18:24:20,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000007 transitioned from NEW to LOCALIZING 2016-11-02 18:24:20,295 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.195 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0150 CONTAINERID=container_e06_1478007721435_0150_01_000007 2016-11-02 18:24:20,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0150 2016-11-02 18:24:20,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0150_01_000007 2016-11-02 18:24:20,295 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:20,296 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0150_01_000007.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:20,296 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000007 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:24:20,298 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:24:20,298 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:24:20,298 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0150 CONTAINERID=container_e06_1478007721435_0150_01_000007 2016-11-02 18:24:20,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0150_01_000007 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:24:20,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0150_01_000007 from application application_1478007721435_0150 2016-11-02 18:24:20,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0150_01_000007 2016-11-02 18:24:20,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0150_01_000007 for log-aggregation 2016-11-02 18:24:20,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0150 2016-11-02 18:24:20,759 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0151_000001 (auth:SIMPLE) 2016-11-02 18:24:20,765 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0151_01_000003 by user hdfs 2016-11-02 18:24:20,766 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0151 2016-11-02 18:24:20,767 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.205 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0151 CONTAINERID=container_e06_1478007721435_0151_01_000003 2016-11-02 18:24:20,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0151 transitioned from NEW to INITING 2016-11-02 18:24:20,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0151_01_000003 to application application_1478007721435_0151 2016-11-02 18:24:20,769 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:24:20,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0151 transitioned from INITING to RUNNING 2016-11-02 18:24:20,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0151_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:24:20,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0151 2016-11-02 18:24:20,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0151_01_000003 2016-11-02 18:24:20,805 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:20,805 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0151_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:24:20,805 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0151_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:24:20,806 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:24:20,806 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:24:20,807 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0151 CONTAINERID=container_e06_1478007721435_0151_01_000003 2016-11-02 18:24:20,808 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0151_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:24:20,808 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0151_01_000003 from application application_1478007721435_0151 2016-11-02 18:24:20,809 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0151_01_000003 2016-11-02 18:24:20,809 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0151_01_000003 for log-aggregation 2016-11-02 18:24:20,809 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0151 2016-11-02 18:24:26,818 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0151_01_000003] 2016-11-02 18:24:38,838 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0151 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:24:38,839 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0151 2016-11-02 18:24:38,839 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0151 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:24:38,839 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0151 2016-11-02 18:24:38,860 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0151_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:25:11,889 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0153_01_000004 2016-11-02 18:25:16,897 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find application application_1478007721435_0153 while processing FINISH_APPS event 2016-11-02 18:25:35,619 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0155_000001 (auth:SIMPLE) 2016-11-02 18:25:35,626 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0155_01_000003 by user hdfs 2016-11-02 18:25:35,628 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0155 2016-11-02 18:25:35,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0155 transitioned from NEW to INITING 2016-11-02 18:25:35,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0155_01_000003 to application application_1478007721435_0155 2016-11-02 18:25:35,629 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.205 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0155 CONTAINERID=container_e06_1478007721435_0155_01_000003 2016-11-02 18:25:35,667 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0155 transitioned from INITING to RUNNING 2016-11-02 18:25:35,667 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0155_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:25:35,667 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0155 2016-11-02 18:25:35,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0155_01_000003 2016-11-02 18:25:35,668 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:25:35,669 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0155_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:25:35,669 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0155_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:25:35,670 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:25:35,671 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:25:35,672 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0155 CONTAINERID=container_e06_1478007721435_0155_01_000003 2016-11-02 18:25:35,673 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0155_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:25:35,673 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0155_01_000003 from application application_1478007721435_0155 2016-11-02 18:25:35,673 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0155_01_000003 2016-11-02 18:25:35,673 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0155_01_000003 for log-aggregation 2016-11-02 18:25:35,673 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0155 2016-11-02 18:25:38,681 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0150_01_000007] 2016-11-02 18:25:42,689 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0155_01_000003] 2016-11-02 18:25:53,705 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0155 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:25:53,706 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0155 2016-11-02 18:25:53,706 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0155 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:25:53,706 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0155 2016-11-02 18:25:53,727 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0155_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:25:54,706 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0150 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:25:54,707 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0150 2016-11-02 18:25:54,707 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0150 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:25:54,707 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0150 2016-11-02 18:25:54,727 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0150_01_000007. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:25:54,728 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0150_01_000005. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:25:54,728 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0150_01_000006. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:25:54,728 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0150_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:25:58,428 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0156_000001 (auth:SIMPLE) 2016-11-02 18:25:58,436 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0156_01_000003 by user hdfs 2016-11-02 18:25:58,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0156 2016-11-02 18:25:58,439 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0156 transitioned from NEW to INITING 2016-11-02 18:25:58,439 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0156_01_000003 to application application_1478007721435_0156 2016-11-02 18:25:58,439 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.205 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0156 CONTAINERID=container_e06_1478007721435_0156_01_000003 2016-11-02 18:25:58,441 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:25:58,475 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0156 transitioned from INITING to RUNNING 2016-11-02 18:25:58,475 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:25:58,476 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0156 2016-11-02 18:25:58,476 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0156_01_000003 2016-11-02 18:25:58,476 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:25:58,476 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0156_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:25:58,477 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:25:58,477 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:25:58,477 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:25:58,477 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0156 CONTAINERID=container_e06_1478007721435_0156_01_000003 2016-11-02 18:25:58,478 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:25:58,478 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0156_01_000003 from application application_1478007721435_0156 2016-11-02 18:25:58,478 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0156_01_000003 2016-11-02 18:25:58,478 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0156_01_000003 for log-aggregation 2016-11-02 18:25:58,478 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0156 2016-11-02 18:26:04,488 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0156_01_000003] 2016-11-02 18:26:04,840 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0156_000001 (auth:SIMPLE) 2016-11-02 18:26:04,852 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0156_01_000004 by user hdfs 2016-11-02 18:26:04,854 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0156_01_000004 to application application_1478007721435_0156 2016-11-02 18:26:04,854 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000004 transitioned from NEW to LOCALIZING 2016-11-02 18:26:04,854 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0156 2016-11-02 18:26:04,854 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.205 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0156 CONTAINERID=container_e06_1478007721435_0156_01_000004 2016-11-02 18:26:04,854 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0156_01_000004 2016-11-02 18:26:04,855 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:04,855 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0156_01_000004.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:04,856 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000004 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:26:04,857 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:26:04,857 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:26:04,858 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0156 CONTAINERID=container_e06_1478007721435_0156_01_000004 2016-11-02 18:26:04,858 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000004 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:26:04,858 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0156_01_000004 from application application_1478007721435_0156 2016-11-02 18:26:04,858 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0156_01_000004 2016-11-02 18:26:04,858 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0156_01_000004 for log-aggregation 2016-11-02 18:26:04,858 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0156 2016-11-02 18:26:10,868 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0156_01_000004] 2016-11-02 18:26:11,044 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0156_000001 (auth:SIMPLE) 2016-11-02 18:26:11,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0156_01_000005 by user hdfs 2016-11-02 18:26:11,050 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0156_01_000005 to application application_1478007721435_0156 2016-11-02 18:26:11,050 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000005 transitioned from NEW to LOCALIZING 2016-11-02 18:26:11,050 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0156 2016-11-02 18:26:11,050 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0156_01_000005 2016-11-02 18:26:11,053 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.205 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0156 CONTAINERID=container_e06_1478007721435_0156_01_000005 2016-11-02 18:26:11,054 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:11,054 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0156_01_000005.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:11,054 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000005 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:26:11,055 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:26:11,055 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:26:11,055 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0156 CONTAINERID=container_e06_1478007721435_0156_01_000005 2016-11-02 18:26:11,056 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0156_01_000005 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:26:11,056 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0156_01_000005 from application application_1478007721435_0156 2016-11-02 18:26:11,056 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0156_01_000005 2016-11-02 18:26:11,056 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0156_01_000005 for log-aggregation 2016-11-02 18:26:11,056 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0156 2016-11-02 18:26:17,065 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0156_01_000005] 2016-11-02 18:26:28,083 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0156 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:26:28,084 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0156 2016-11-02 18:26:28,084 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0156 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:26:28,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0156 2016-11-02 18:26:28,106 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0156_01_000005. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:26:28,106 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0156_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:26:28,106 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0156_01_000004. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:26:32,084 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0159_000001 (auth:SIMPLE) 2016-11-02 18:26:32,091 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0159_01_000003 by user hdfs 2016-11-02 18:26:32,093 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0159 2016-11-02 18:26:32,093 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0159 transitioned from NEW to INITING 2016-11-02 18:26:32,093 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0159_01_000003 to application application_1478007721435_0159 2016-11-02 18:26:32,094 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.198 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0159 CONTAINERID=container_e06_1478007721435_0159_01_000003 2016-11-02 18:26:32,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0159 transitioned from INITING to RUNNING 2016-11-02 18:26:32,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0159_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:26:32,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0159 2016-11-02 18:26:32,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0159_01_000003 2016-11-02 18:26:32,134 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:32,134 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0159_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:32,134 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0159_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:26:32,136 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:26:32,136 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:26:32,136 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0159 CONTAINERID=container_e06_1478007721435_0159_01_000003 2016-11-02 18:26:32,136 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0159_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:26:32,136 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0159_01_000003 from application application_1478007721435_0159 2016-11-02 18:26:32,136 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0159_01_000003 2016-11-02 18:26:32,136 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0159_01_000003 for log-aggregation 2016-11-02 18:26:32,137 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0159 2016-11-02 18:26:37,518 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0160_000001 (auth:SIMPLE) 2016-11-02 18:26:37,526 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0160_01_000003 by user hdfs 2016-11-02 18:26:37,528 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0160 2016-11-02 18:26:37,529 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0160 transitioned from NEW to INITING 2016-11-02 18:26:37,529 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0160_01_000003 to application application_1478007721435_0160 2016-11-02 18:26:37,529 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.205 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0160 CONTAINERID=container_e06_1478007721435_0160_01_000003 2016-11-02 18:26:37,532 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:26:37,568 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0160 transitioned from INITING to RUNNING 2016-11-02 18:26:37,568 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:26:37,568 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0160 2016-11-02 18:26:37,569 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0160_01_000003 2016-11-02 18:26:37,570 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:37,570 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0160_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:37,570 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:26:37,571 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:26:37,571 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:26:37,571 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0160 CONTAINERID=container_e06_1478007721435_0160_01_000003 2016-11-02 18:26:37,572 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:26:37,572 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0160_01_000003 from application application_1478007721435_0160 2016-11-02 18:26:37,572 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0160_01_000003 2016-11-02 18:26:37,572 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0160_01_000003 for log-aggregation 2016-11-02 18:26:37,572 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0160 2016-11-02 18:26:38,575 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0159_01_000003] 2016-11-02 18:26:38,575 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0159_01_000004 2016-11-02 18:26:43,582 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0160_01_000003] 2016-11-02 18:26:43,966 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0160_000001 (auth:SIMPLE) 2016-11-02 18:26:43,971 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0160_01_000004 by user hdfs 2016-11-02 18:26:43,973 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0160_01_000004 to application application_1478007721435_0160 2016-11-02 18:26:43,973 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000004 transitioned from NEW to LOCALIZING 2016-11-02 18:26:43,973 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0160 2016-11-02 18:26:43,973 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.205 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0160 CONTAINERID=container_e06_1478007721435_0160_01_000004 2016-11-02 18:26:43,973 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0160_01_000004 2016-11-02 18:26:43,974 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:43,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0160_01_000004.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:43,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000004 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:26:43,976 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:26:43,976 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:26:43,976 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0160 CONTAINERID=container_e06_1478007721435_0160_01_000004 2016-11-02 18:26:43,977 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000004 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:26:43,977 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0160_01_000004 from application application_1478007721435_0160 2016-11-02 18:26:43,977 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0160_01_000004 2016-11-02 18:26:43,977 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0160_01_000004 for log-aggregation 2016-11-02 18:26:43,977 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0160 2016-11-02 18:26:47,996 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0161_000001 (auth:SIMPLE) 2016-11-02 18:26:48,000 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0161_01_000001 by user hdfs 2016-11-02 18:26:48,002 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0161 2016-11-02 18:26:48,002 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0161 transitioned from NEW to INITING 2016-11-02 18:26:48,002 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0161_01_000001 to application application_1478007721435_0161 2016-11-02 18:26:48,003 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0161 CONTAINERID=container_e06_1478007721435_0161_01_000001 2016-11-02 18:26:48,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0161 transitioned from INITING to RUNNING 2016-11-02 18:26:48,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0161_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:26:48,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0161 2016-11-02 18:26:48,042 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0161_01_000001 2016-11-02 18:26:48,042 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:48,042 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0161_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:48,043 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0161_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:26:48,043 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:26:48,043 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:26:48,045 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0161 CONTAINERID=container_e06_1478007721435_0161_01_000001 2016-11-02 18:26:48,045 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0161_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:26:48,045 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0161_01_000001 from application application_1478007721435_0161 2016-11-02 18:26:48,045 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0161_01_000001 2016-11-02 18:26:48,045 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0161_01_000001 for log-aggregation 2016-11-02 18:26:48,045 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0161 2016-11-02 18:26:49,048 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0161_01_000001] 2016-11-02 18:26:49,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0161 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:26:49,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0161 2016-11-02 18:26:49,049 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0161 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:26:49,049 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0161 2016-11-02 18:26:49,067 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0161_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:26:50,049 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0160_01_000004] 2016-11-02 18:26:50,050 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0159 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:26:50,050 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0159 2016-11-02 18:26:50,050 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0159 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:26:50,050 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0159 2016-11-02 18:26:50,070 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0159_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:26:50,171 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0160_000001 (auth:SIMPLE) 2016-11-02 18:26:50,176 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0160_01_000005 by user hdfs 2016-11-02 18:26:50,180 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0160_01_000005 to application application_1478007721435_0160 2016-11-02 18:26:50,180 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000005 transitioned from NEW to LOCALIZING 2016-11-02 18:26:50,180 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0160 2016-11-02 18:26:50,180 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.205 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0160 CONTAINERID=container_e06_1478007721435_0160_01_000005 2016-11-02 18:26:50,180 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0160_01_000005 2016-11-02 18:26:50,181 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:50,181 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0160_01_000005.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:26:50,181 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000005 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:26:50,182 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:26:50,182 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:26:50,182 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0160 CONTAINERID=container_e06_1478007721435_0160_01_000005 2016-11-02 18:26:50,183 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0160_01_000005 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:26:50,183 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0160_01_000005 from application application_1478007721435_0160 2016-11-02 18:26:50,183 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0160_01_000005 2016-11-02 18:26:50,183 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0160_01_000005 for log-aggregation 2016-11-02 18:26:50,183 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0160 2016-11-02 18:26:56,191 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0160_01_000005] 2016-11-02 18:27:02,200 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0160 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:27:02,200 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0160 2016-11-02 18:27:02,201 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0160 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:27:02,201 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0160 2016-11-02 18:27:02,222 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0160_01_000004. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:27:02,222 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0160_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:27:02,222 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0160_01_000005. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:27:38,271 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0162_000002 (auth:SIMPLE) 2016-11-02 18:27:38,277 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0162_02_000001 by user hdfs 2016-11-02 18:27:38,280 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0162 2016-11-02 18:27:38,281 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0162 transitioned from NEW to INITING 2016-11-02 18:27:38,281 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0162_02_000001 to application application_1478007721435_0162 2016-11-02 18:27:38,281 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0162 CONTAINERID=container_e06_1478007721435_0162_02_000001 2016-11-02 18:27:38,285 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:27:38,317 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0162 transitioned from INITING to RUNNING 2016-11-02 18:27:38,318 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0162_02_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:27:38,318 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0162 2016-11-02 18:27:38,319 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0162_02_000001 2016-11-02 18:27:38,320 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:27:38,321 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0162_02_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:27:38,323 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0162_02_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:27:38,323 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:27:38,324 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:27:38,324 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0162 CONTAINERID=container_e06_1478007721435_0162_02_000001 2016-11-02 18:27:38,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0162_02_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:27:38,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0162_02_000001 from application application_1478007721435_0162 2016-11-02 18:27:38,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0162_02_000001 2016-11-02 18:27:38,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0162_02_000001 for log-aggregation 2016-11-02 18:27:38,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0162 2016-11-02 18:27:39,327 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0162_02_000001] 2016-11-02 18:27:39,327 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0162 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:27:39,328 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0162 2016-11-02 18:27:39,328 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0162 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:27:39,328 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0162 2016-11-02 18:27:39,348 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0162_02_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:45:38,968 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0165_000001 (auth:SIMPLE) 2016-11-02 18:45:38,976 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0165_01_000001 by user hdfs 2016-11-02 18:45:38,982 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0165 2016-11-02 18:45:38,983 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0165 transitioned from NEW to INITING 2016-11-02 18:45:38,983 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0165_01_000001 to application application_1478007721435_0165 2016-11-02 18:45:38,985 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0165 CONTAINERID=container_e06_1478007721435_0165_01_000001 2016-11-02 18:45:39,040 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0165 transitioned from INITING to RUNNING 2016-11-02 18:45:39,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0165_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:45:39,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0165 2016-11-02 18:45:39,042 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0165_01_000001 2016-11-02 18:45:39,042 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:45:39,043 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0165_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:45:39,043 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0165_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:45:39,045 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:45:39,045 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:45:39,045 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0165 CONTAINERID=container_e06_1478007721435_0165_01_000001 2016-11-02 18:45:39,046 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0165_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:45:39,046 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0165_01_000001 from application application_1478007721435_0165 2016-11-02 18:45:39,046 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0165_01_000001 2016-11-02 18:45:39,046 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0165_01_000001 for log-aggregation 2016-11-02 18:45:39,046 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0165 2016-11-02 18:45:40,052 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0165_01_000001] 2016-11-02 18:45:40,055 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0165 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:45:40,055 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0165 2016-11-02 18:45:40,056 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0165 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:45:40,057 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0165 2016-11-02 18:45:40,074 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0165_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:45:54,086 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0167_000001 (auth:SIMPLE) 2016-11-02 18:45:54,091 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0167_01_000001 by user hdfs 2016-11-02 18:45:54,092 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0167 2016-11-02 18:45:54,095 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0167 transitioned from NEW to INITING 2016-11-02 18:45:54,095 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0167_01_000001 to application application_1478007721435_0167 2016-11-02 18:45:54,096 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0167 CONTAINERID=container_e06_1478007721435_0167_01_000001 2016-11-02 18:45:54,098 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:45:54,146 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0167 transitioned from INITING to RUNNING 2016-11-02 18:45:54,146 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0167_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:45:54,146 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0167 2016-11-02 18:45:54,147 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0167_01_000001 2016-11-02 18:45:54,147 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:45:54,148 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0167_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:45:54,148 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0167_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:45:54,149 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:45:54,149 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:45:54,149 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0167 CONTAINERID=container_e06_1478007721435_0167_01_000001 2016-11-02 18:45:54,149 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0167_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:45:54,149 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0167_01_000001 from application application_1478007721435_0167 2016-11-02 18:45:54,150 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0167_01_000001 2016-11-02 18:45:54,150 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0167_01_000001 for log-aggregation 2016-11-02 18:45:54,150 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0167 2016-11-02 18:45:55,152 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0167_01_000001] 2016-11-02 18:45:55,152 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0167 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:45:55,153 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0167 2016-11-02 18:45:55,153 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0167 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:45:55,153 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0167 2016-11-02 18:45:55,183 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0167_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:45:56,167 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0170_000001 (auth:SIMPLE) 2016-11-02 18:45:56,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0170_01_000001 by user hdfs 2016-11-02 18:45:56,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0170 2016-11-02 18:45:56,173 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0170 transitioned from NEW to INITING 2016-11-02 18:45:56,173 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0170_01_000001 to application application_1478007721435_0170 2016-11-02 18:45:56,175 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0170 CONTAINERID=container_e06_1478007721435_0170_01_000001 2016-11-02 18:45:56,213 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0170 transitioned from INITING to RUNNING 2016-11-02 18:45:56,213 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0170_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:45:56,213 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0170 2016-11-02 18:45:56,214 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0170_01_000001 2016-11-02 18:45:56,214 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:45:56,215 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0170_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:45:56,215 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:45:56,215 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:45:56,216 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0170_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:45:56,216 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0170 CONTAINERID=container_e06_1478007721435_0170_01_000001 2016-11-02 18:45:56,218 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0170_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:45:56,218 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0170_01_000001 from application application_1478007721435_0170 2016-11-02 18:45:56,218 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0170_01_000001 2016-11-02 18:45:56,218 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0170_01_000001 for log-aggregation 2016-11-02 18:45:56,218 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0170 2016-11-02 18:45:57,220 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0170_01_000001] 2016-11-02 18:45:57,221 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0170 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:45:57,221 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0170 2016-11-02 18:45:57,221 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0170 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:45:57,221 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0170 2016-11-02 18:45:57,244 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0170_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:46:08,036 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0166_000001 (auth:SIMPLE) 2016-11-02 18:46:08,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0166_01_000003 by user hdfs 2016-11-02 18:46:08,042 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0166 2016-11-02 18:46:08,043 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0166 transitioned from NEW to INITING 2016-11-02 18:46:08,043 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0166_01_000003 to application application_1478007721435_0166 2016-11-02 18:46:08,043 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0166 CONTAINERID=container_e06_1478007721435_0166_01_000003 2016-11-02 18:46:08,050 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:46:08,084 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0166 transitioned from INITING to RUNNING 2016-11-02 18:46:08,084 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0166_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:46:08,084 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0166 2016-11-02 18:46:08,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0166_01_000003 2016-11-02 18:46:08,085 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:46:08,086 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0166_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:46:08,086 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0166_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:46:08,087 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:46:08,087 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:46:08,087 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0166 CONTAINERID=container_e06_1478007721435_0166_01_000003 2016-11-02 18:46:08,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0166_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:46:08,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0166_01_000003 from application application_1478007721435_0166 2016-11-02 18:46:08,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0166_01_000003 2016-11-02 18:46:08,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0166_01_000003 for log-aggregation 2016-11-02 18:46:08,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0166 2016-11-02 18:46:13,567 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0169_000001 (auth:SIMPLE) 2016-11-02 18:46:13,682 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0169_01_000002 by user hdfs 2016-11-02 18:46:13,683 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0169 2016-11-02 18:46:13,684 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0169 transitioned from NEW to INITING 2016-11-02 18:46:13,684 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0169_01_000002 to application application_1478007721435_0169 2016-11-02 18:46:13,684 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0169 CONTAINERID=container_e06_1478007721435_0169_01_000002 2016-11-02 18:46:13,722 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0169 transitioned from INITING to RUNNING 2016-11-02 18:46:13,722 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0169_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:46:13,722 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0169 2016-11-02 18:46:13,722 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0169_01_000002 2016-11-02 18:46:13,723 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:46:13,723 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0169_01_000002.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:46:13,723 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0169_01_000002 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:46:13,724 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:46:13,724 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:46:13,724 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0169 CONTAINERID=container_e06_1478007721435_0169_01_000002 2016-11-02 18:46:13,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0169_01_000002 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:46:13,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0169_01_000002 from application application_1478007721435_0169 2016-11-02 18:46:13,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0169_01_000002 2016-11-02 18:46:13,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0169_01_000002 for log-aggregation 2016-11-02 18:46:13,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0169 2016-11-02 18:46:14,728 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0166_01_000003] 2016-11-02 18:46:17,733 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0169_01_000002] 2016-11-02 18:46:18,094 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0169_000001 (auth:SIMPLE) 2016-11-02 18:46:18,105 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0169_01_000004 by user hdfs 2016-11-02 18:46:18,107 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0169_01_000004 to application application_1478007721435_0169 2016-11-02 18:46:18,107 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0169_01_000004 transitioned from NEW to LOCALIZING 2016-11-02 18:46:18,107 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0169 2016-11-02 18:46:18,107 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0169 CONTAINERID=container_e06_1478007721435_0169_01_000004 2016-11-02 18:46:18,107 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0169_01_000004 2016-11-02 18:46:18,108 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:46:18,108 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0169_01_000004.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:46:18,108 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0169_01_000004 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:46:18,109 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:46:18,109 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:46:18,109 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0169 CONTAINERID=container_e06_1478007721435_0169_01_000004 2016-11-02 18:46:18,110 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0169_01_000004 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:46:18,110 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0169_01_000004 from application application_1478007721435_0169 2016-11-02 18:46:18,110 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0169_01_000004 2016-11-02 18:46:18,110 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0169_01_000004 for log-aggregation 2016-11-02 18:46:18,110 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0169 2016-11-02 18:46:37,211 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0172_000001 (auth:SIMPLE) 2016-11-02 18:46:37,677 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0172_01_000006 by user hdfs 2016-11-02 18:46:37,678 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0172 2016-11-02 18:46:37,679 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0172 transitioned from NEW to INITING 2016-11-02 18:46:37,679 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0172_01_000006 to application application_1478007721435_0172 2016-11-02 18:46:37,680 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0172 CONTAINERID=container_e06_1478007721435_0172_01_000006 2016-11-02 18:46:37,683 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:46:37,723 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0172 transitioned from INITING to RUNNING 2016-11-02 18:46:37,723 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0172_01_000006 transitioned from NEW to LOCALIZING 2016-11-02 18:46:37,723 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0172 2016-11-02 18:46:37,723 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0172_01_000006 2016-11-02 18:46:37,724 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:46:37,724 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0172_01_000006.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:46:37,725 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0172_01_000006 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:46:37,725 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:46:37,725 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:46:37,725 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0172 CONTAINERID=container_e06_1478007721435_0172_01_000006 2016-11-02 18:46:37,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0172_01_000006 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:46:37,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0172_01_000006 from application application_1478007721435_0172 2016-11-02 18:46:37,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0172_01_000006 2016-11-02 18:46:37,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0172_01_000006 for log-aggregation 2016-11-02 18:46:37,726 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0172 2016-11-02 18:48:28,913 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0178_000001 (auth:SIMPLE) 2016-11-02 18:48:28,923 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0178_01_000001 by user hdfs 2016-11-02 18:48:28,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0178 2016-11-02 18:48:28,930 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0178 transitioned from NEW to INITING 2016-11-02 18:48:28,930 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0178_01_000001 to application application_1478007721435_0178 2016-11-02 18:48:28,931 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0178 CONTAINERID=container_e06_1478007721435_0178_01_000001 2016-11-02 18:48:28,984 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0178 transitioned from INITING to RUNNING 2016-11-02 18:48:28,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0178_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:48:28,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0178 2016-11-02 18:48:28,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0178_01_000001 2016-11-02 18:48:28,986 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:48:28,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0178_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:48:28,987 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0178_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:48:28,987 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:48:28,988 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:48:28,988 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0178 CONTAINERID=container_e06_1478007721435_0178_01_000001 2016-11-02 18:48:28,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0178_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:48:28,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0178_01_000001 from application application_1478007721435_0178 2016-11-02 18:48:28,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0178_01_000001 2016-11-02 18:48:28,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0178_01_000001 for log-aggregation 2016-11-02 18:48:28,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0178 2016-11-02 18:48:29,992 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0178_01_000001] 2016-11-02 18:48:29,993 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0178 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:48:29,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0178 2016-11-02 18:48:29,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0178 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:48:29,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0178 2016-11-02 18:48:30,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0178_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:49:11,072 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0186_000001 (auth:SIMPLE) 2016-11-02 18:49:11,078 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0186_01_000001 by user hdfs 2016-11-02 18:49:11,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0186 2016-11-02 18:49:11,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0186 transitioned from NEW to INITING 2016-11-02 18:49:11,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0186_01_000001 to application application_1478007721435_0186 2016-11-02 18:49:11,081 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0186 CONTAINERID=container_e06_1478007721435_0186_01_000001 2016-11-02 18:49:11,084 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:49:11,125 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0186 transitioned from INITING to RUNNING 2016-11-02 18:49:11,125 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0186_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:49:11,125 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0186 2016-11-02 18:49:11,126 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0186_01_000001 2016-11-02 18:49:11,127 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:11,127 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0186_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:11,127 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0186_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:49:11,129 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:49:11,129 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:49:11,129 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0186 CONTAINERID=container_e06_1478007721435_0186_01_000001 2016-11-02 18:49:11,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0186_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:49:11,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0186_01_000001 from application application_1478007721435_0186 2016-11-02 18:49:11,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0186_01_000001 2016-11-02 18:49:11,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0186_01_000001 for log-aggregation 2016-11-02 18:49:11,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0186 2016-11-02 18:49:12,133 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0186_01_000001] 2016-11-02 18:49:12,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0186 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:49:12,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0186 2016-11-02 18:49:12,134 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0186 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:49:12,134 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0186 2016-11-02 18:49:12,158 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0186_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:49:18,691 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0183_000001 (auth:SIMPLE) 2016-11-02 18:49:19,152 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0183_01_000002 by user hdfs 2016-11-02 18:49:19,153 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0183 2016-11-02 18:49:19,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0183 transitioned from NEW to INITING 2016-11-02 18:49:19,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0183_01_000002 to application application_1478007721435_0183 2016-11-02 18:49:19,156 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0183 CONTAINERID=container_e06_1478007721435_0183_01_000002 2016-11-02 18:49:19,207 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0183 transitioned from INITING to RUNNING 2016-11-02 18:49:19,207 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0183_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:49:19,207 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0183 2016-11-02 18:49:19,208 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0183_01_000002 2016-11-02 18:49:19,208 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:19,208 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0183_01_000002.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:19,210 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0183_01_000002 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:49:19,212 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:49:19,213 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:49:19,213 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0183 CONTAINERID=container_e06_1478007721435_0183_01_000002 2016-11-02 18:49:19,214 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0183_01_000002 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:49:19,214 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0183_01_000002 from application application_1478007721435_0183 2016-11-02 18:49:19,214 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0183_01_000002 2016-11-02 18:49:19,214 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0183_01_000002 for log-aggregation 2016-11-02 18:49:19,214 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0183 2016-11-02 18:49:22,232 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0187_000001 (auth:SIMPLE) 2016-11-02 18:49:22,239 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0187_01_000001 by user hdfs 2016-11-02 18:49:22,244 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0187 2016-11-02 18:49:22,245 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0187 transitioned from NEW to INITING 2016-11-02 18:49:22,246 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0187 CONTAINERID=container_e06_1478007721435_0187_01_000001 2016-11-02 18:49:22,246 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:49:22,287 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0187_01_000001 to application application_1478007721435_0187 2016-11-02 18:49:22,287 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0187 transitioned from INITING to RUNNING 2016-11-02 18:49:22,288 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0187_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:49:22,288 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0187 2016-11-02 18:49:22,288 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0187_01_000001 2016-11-02 18:49:22,288 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:22,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0187_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:22,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0187_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:49:22,289 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:49:22,289 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:49:22,290 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0187 CONTAINERID=container_e06_1478007721435_0187_01_000001 2016-11-02 18:49:22,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0187_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:49:22,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0187_01_000001 from application application_1478007721435_0187 2016-11-02 18:49:22,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0187_01_000001 2016-11-02 18:49:22,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0187_01_000001 for log-aggregation 2016-11-02 18:49:22,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0187 2016-11-02 18:49:22,305 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0188_000001 (auth:SIMPLE) 2016-11-02 18:49:22,310 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0188_01_000001 by user hdfs 2016-11-02 18:49:22,311 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0188 2016-11-02 18:49:22,311 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0188 transitioned from NEW to INITING 2016-11-02 18:49:22,311 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0188_01_000001 to application application_1478007721435_0188 2016-11-02 18:49:22,313 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0188 CONTAINERID=container_e06_1478007721435_0188_01_000001 2016-11-02 18:49:22,354 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0188 transitioned from INITING to RUNNING 2016-11-02 18:49:22,354 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0188_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:49:22,354 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0188 2016-11-02 18:49:22,355 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0188_01_000001 2016-11-02 18:49:22,355 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:22,356 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0188_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:22,356 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0188_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:49:22,357 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:49:22,357 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:49:22,357 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0188 CONTAINERID=container_e06_1478007721435_0188_01_000001 2016-11-02 18:49:22,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0188_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:49:22,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0188_01_000001 from application application_1478007721435_0188 2016-11-02 18:49:22,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0188_01_000001 2016-11-02 18:49:22,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0188_01_000001 for log-aggregation 2016-11-02 18:49:22,358 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0188 2016-11-02 18:49:22,359 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0187_01_000001] 2016-11-02 18:49:22,359 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0187 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:49:22,359 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0187 2016-11-02 18:49:22,360 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0187 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:49:22,360 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0187 2016-11-02 18:49:22,381 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0187_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:49:23,361 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0188_01_000001] 2016-11-02 18:49:23,361 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0188 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:49:23,361 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0188 2016-11-02 18:49:23,361 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0188 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:49:23,361 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0188 2016-11-02 18:49:23,382 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0188_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:49:25,363 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0183_01_000002] 2016-11-02 18:49:25,364 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0183_01_000004 2016-11-02 18:49:26,116 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0185_000001 (auth:SIMPLE) 2016-11-02 18:49:26,415 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0185_01_000003 by user hdfs 2016-11-02 18:49:26,417 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0185 2016-11-02 18:49:26,417 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0185 transitioned from NEW to INITING 2016-11-02 18:49:26,417 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0185_01_000003 to application application_1478007721435_0185 2016-11-02 18:49:26,417 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0185 CONTAINERID=container_e06_1478007721435_0185_01_000003 2016-11-02 18:49:26,418 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:49:26,460 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0185 transitioned from INITING to RUNNING 2016-11-02 18:49:26,460 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0185_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:49:26,460 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0185 2016-11-02 18:49:26,461 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0185_01_000003 2016-11-02 18:49:26,462 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:26,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0185_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:26,463 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:49:26,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0185_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:49:26,463 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:49:26,465 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0185 CONTAINERID=container_e06_1478007721435_0185_01_000003 2016-11-02 18:49:26,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0185_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:49:26,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0185_01_000003 from application application_1478007721435_0185 2016-11-02 18:49:26,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0185_01_000003 2016-11-02 18:49:26,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0185_01_000003 for log-aggregation 2016-11-02 18:49:26,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0185 2016-11-02 18:49:31,473 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0185_01_000003] 2016-11-02 18:49:45,514 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0190_000001 (auth:SIMPLE) 2016-11-02 18:49:45,520 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0190_01_000001 by user hdfs 2016-11-02 18:49:45,522 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0190 2016-11-02 18:49:45,523 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0190 transitioned from NEW to INITING 2016-11-02 18:49:45,523 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0190_01_000001 to application application_1478007721435_0190 2016-11-02 18:49:45,525 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0190 CONTAINERID=container_e06_1478007721435_0190_01_000001 2016-11-02 18:49:45,569 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0190 transitioned from INITING to RUNNING 2016-11-02 18:49:45,569 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0190_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:49:45,570 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0190 2016-11-02 18:49:45,570 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0190_01_000001 2016-11-02 18:49:45,571 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:45,571 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0190_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:49:45,572 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0190_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:49:45,572 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:49:45,572 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:49:45,573 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0190 CONTAINERID=container_e06_1478007721435_0190_01_000001 2016-11-02 18:49:45,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0190_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:49:45,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0190_01_000001 from application application_1478007721435_0190 2016-11-02 18:49:45,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0190_01_000001 2016-11-02 18:49:45,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0190_01_000001 for log-aggregation 2016-11-02 18:49:45,574 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0190 2016-11-02 18:49:46,577 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0190_01_000001] 2016-11-02 18:49:46,577 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0190 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:49:46,577 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0190 2016-11-02 18:49:46,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0190 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:49:46,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0190 2016-11-02 18:49:46,594 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0190_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:50:41,689 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0192_000001 (auth:SIMPLE) 2016-11-02 18:50:41,695 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0192_01_000001 by user hdfs 2016-11-02 18:50:41,697 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0192 2016-11-02 18:50:41,698 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0192 transitioned from NEW to INITING 2016-11-02 18:50:41,701 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0192 CONTAINERID=container_e06_1478007721435_0192_01_000001 2016-11-02 18:50:41,702 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:50:41,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0192_01_000001 to application application_1478007721435_0192 2016-11-02 18:50:41,746 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0192 transitioned from INITING to RUNNING 2016-11-02 18:50:41,746 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0192_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:50:41,746 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0192 2016-11-02 18:50:41,746 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0192_01_000001 2016-11-02 18:50:41,747 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:50:41,747 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0192_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:50:41,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0192_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:50:41,749 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:50:41,749 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:50:41,750 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0192 CONTAINERID=container_e06_1478007721435_0192_01_000001 2016-11-02 18:50:41,750 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0192_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:50:41,750 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0192_01_000001 from application application_1478007721435_0192 2016-11-02 18:50:41,751 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0192_01_000001 2016-11-02 18:50:41,751 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0192_01_000001 for log-aggregation 2016-11-02 18:50:41,751 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0192 2016-11-02 18:50:42,753 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0192_01_000001] 2016-11-02 18:50:42,755 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0192 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:50:42,755 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0192 2016-11-02 18:50:42,757 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0192 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:50:42,757 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0192 2016-11-02 18:50:42,776 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0192_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:50:49,782 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0193_000001 (auth:SIMPLE) 2016-11-02 18:50:49,787 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0193_01_000001 by user hdfs 2016-11-02 18:50:49,789 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0193 2016-11-02 18:50:49,789 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0193 transitioned from NEW to INITING 2016-11-02 18:50:49,789 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0193_01_000001 to application application_1478007721435_0193 2016-11-02 18:50:49,790 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0193 CONTAINERID=container_e06_1478007721435_0193_01_000001 2016-11-02 18:50:49,833 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0193 transitioned from INITING to RUNNING 2016-11-02 18:50:49,833 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0193_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:50:49,833 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0193 2016-11-02 18:50:49,834 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0193_01_000001 2016-11-02 18:50:49,834 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:50:49,834 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0193_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:50:49,835 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0193_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:50:49,836 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:50:49,836 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:50:49,837 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0193 CONTAINERID=container_e06_1478007721435_0193_01_000001 2016-11-02 18:50:49,838 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0193_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:50:49,838 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0193_01_000001 from application application_1478007721435_0193 2016-11-02 18:50:49,838 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0193_01_000001 2016-11-02 18:50:49,838 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0193_01_000001 for log-aggregation 2016-11-02 18:50:49,838 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0193 2016-11-02 18:50:50,840 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0193_01_000001] 2016-11-02 18:50:50,842 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0193 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:50:50,843 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0193 2016-11-02 18:50:50,844 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0193 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:50:50,844 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0193 2016-11-02 18:50:50,863 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0193_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:52:32,010 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0200_000001 (auth:SIMPLE) 2016-11-02 18:52:32,023 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0200_01_000001 by user hdfs 2016-11-02 18:52:32,028 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0200 2016-11-02 18:52:32,029 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0200 transitioned from NEW to INITING 2016-11-02 18:52:32,029 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0200_01_000001 to application application_1478007721435_0200 2016-11-02 18:52:32,030 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0200 CONTAINERID=container_e06_1478007721435_0200_01_000001 2016-11-02 18:52:32,036 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:52:32,091 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0200 transitioned from INITING to RUNNING 2016-11-02 18:52:32,092 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0200_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:52:32,092 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0200 2016-11-02 18:52:32,092 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0200_01_000001 2016-11-02 18:52:32,093 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:52:32,093 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0200_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:52:32,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0200_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:52:32,095 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:52:32,095 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:52:32,095 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0200 CONTAINERID=container_e06_1478007721435_0200_01_000001 2016-11-02 18:52:32,096 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0200_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:52:32,097 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0200_01_000001 from application application_1478007721435_0200 2016-11-02 18:52:32,097 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0200_01_000001 2016-11-02 18:52:32,097 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0200_01_000001 for log-aggregation 2016-11-02 18:52:32,097 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0200 2016-11-02 18:52:33,099 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0200_01_000001] 2016-11-02 18:52:33,099 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0200 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:52:33,100 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0200 2016-11-02 18:52:33,103 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0200 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:52:33,103 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0200 2016-11-02 18:52:33,124 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0200_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:53:34,204 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0205_000001 (auth:SIMPLE) 2016-11-02 18:53:34,212 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0205_01_000001 by user hdfs 2016-11-02 18:53:34,219 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0205 2016-11-02 18:53:34,220 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0205 transitioned from NEW to INITING 2016-11-02 18:53:34,220 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0205_01_000001 to application application_1478007721435_0205 2016-11-02 18:53:34,222 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0205 CONTAINERID=container_e06_1478007721435_0205_01_000001 2016-11-02 18:53:34,294 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0205 transitioned from INITING to RUNNING 2016-11-02 18:53:34,294 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0205_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:53:34,294 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0205 2016-11-02 18:53:34,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0205_01_000001 2016-11-02 18:53:34,296 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:53:34,296 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0205_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:53:34,296 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0205_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:53:34,297 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:53:34,297 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:53:34,298 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0205 CONTAINERID=container_e06_1478007721435_0205_01_000001 2016-11-02 18:53:34,298 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0205_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:53:34,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0205_01_000001 from application application_1478007721435_0205 2016-11-02 18:53:34,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0205_01_000001 2016-11-02 18:53:34,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0205_01_000001 for log-aggregation 2016-11-02 18:53:34,299 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0205 2016-11-02 18:53:35,302 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0205_01_000001] 2016-11-02 18:53:35,305 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0205 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:53:35,306 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0205 2016-11-02 18:53:35,306 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0205 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:53:35,306 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0205 2016-11-02 18:53:35,330 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0205_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:54:44,031 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0206_000001 (auth:SIMPLE) 2016-11-02 18:54:44,038 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0206_01_000003 by user hdfs 2016-11-02 18:54:44,040 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0206 2016-11-02 18:54:44,040 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0206 transitioned from NEW to INITING 2016-11-02 18:54:44,040 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0206_01_000003 to application application_1478007721435_0206 2016-11-02 18:54:44,041 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0206 CONTAINERID=container_e06_1478007721435_0206_01_000003 2016-11-02 18:54:44,045 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:54:44,084 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0206 transitioned from INITING to RUNNING 2016-11-02 18:54:44,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0206_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:54:44,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0206 2016-11-02 18:54:44,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0206_01_000003 2016-11-02 18:54:44,085 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:54:44,086 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0206_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:54:44,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0206_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:54:44,088 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:54:44,088 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:54:44,089 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0206 CONTAINERID=container_e06_1478007721435_0206_01_000003 2016-11-02 18:54:44,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0206_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:54:44,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0206_01_000003 from application application_1478007721435_0206 2016-11-02 18:54:44,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0206_01_000003 2016-11-02 18:54:44,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0206_01_000003 for log-aggregation 2016-11-02 18:54:44,091 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0206 2016-11-02 18:54:50,101 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0206_01_000003] 2016-11-02 18:54:50,457 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0206_000001 (auth:SIMPLE) 2016-11-02 18:54:50,460 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0206_01_000005 by user hdfs 2016-11-02 18:54:50,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0206_01_000005 to application application_1478007721435_0206 2016-11-02 18:54:50,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0206_01_000005 transitioned from NEW to LOCALIZING 2016-11-02 18:54:50,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0206 2016-11-02 18:54:50,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0206_01_000005 2016-11-02 18:54:50,463 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0206 CONTAINERID=container_e06_1478007721435_0206_01_000005 2016-11-02 18:54:50,464 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:54:50,464 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0206_01_000005.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:54:50,464 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0206_01_000005 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:54:50,465 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:54:50,465 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:54:50,465 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0206 CONTAINERID=container_e06_1478007721435_0206_01_000005 2016-11-02 18:54:50,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0206_01_000005 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:54:50,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0206_01_000005 from application application_1478007721435_0206 2016-11-02 18:54:50,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0206_01_000005 2016-11-02 18:54:50,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0206_01_000005 for log-aggregation 2016-11-02 18:54:50,466 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0206 2016-11-02 18:54:57,477 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0206_01_000005] 2016-11-02 18:54:59,491 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0211_000001 (auth:SIMPLE) 2016-11-02 18:54:59,496 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0211_01_000001 by user hdfs 2016-11-02 18:54:59,497 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0211 2016-11-02 18:54:59,498 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0211 transitioned from NEW to INITING 2016-11-02 18:54:59,498 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0211_01_000001 to application application_1478007721435_0211 2016-11-02 18:54:59,498 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0211 CONTAINERID=container_e06_1478007721435_0211_01_000001 2016-11-02 18:54:59,541 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0211 transitioned from INITING to RUNNING 2016-11-02 18:54:59,542 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0211_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:54:59,542 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0211 2016-11-02 18:54:59,542 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0211_01_000001 2016-11-02 18:54:59,542 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:54:59,543 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0211_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:54:59,543 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0211_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:54:59,546 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:54:59,546 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:54:59,546 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0211 CONTAINERID=container_e06_1478007721435_0211_01_000001 2016-11-02 18:54:59,547 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0211_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:54:59,547 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0211_01_000001 from application application_1478007721435_0211 2016-11-02 18:54:59,547 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0211_01_000001 2016-11-02 18:54:59,547 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0211_01_000001 for log-aggregation 2016-11-02 18:54:59,547 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0211 2016-11-02 18:55:00,549 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0211_01_000001] 2016-11-02 18:55:00,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0211 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:55:00,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0211 2016-11-02 18:55:00,553 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0211 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:55:00,553 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0211 2016-11-02 18:55:00,573 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0211_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:55:04,568 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0212_000001 (auth:SIMPLE) 2016-11-02 18:55:04,572 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0212_01_000001 by user hdfs 2016-11-02 18:55:04,575 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0212 2016-11-02 18:55:04,576 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0212 CONTAINERID=container_e06_1478007721435_0212_01_000001 2016-11-02 18:55:04,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0212 transitioned from NEW to INITING 2016-11-02 18:55:04,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0212_01_000001 to application application_1478007721435_0212 2016-11-02 18:55:04,580 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:55:04,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0212 transitioned from INITING to RUNNING 2016-11-02 18:55:04,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0212_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:55:04,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0212 2016-11-02 18:55:04,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0212_01_000001 2016-11-02 18:55:04,630 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:55:04,630 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0212_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:55:04,630 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0212_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:55:04,631 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:55:04,631 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:55:04,631 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0212 CONTAINERID=container_e06_1478007721435_0212_01_000001 2016-11-02 18:55:04,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0212_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:55:04,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0212_01_000001 from application application_1478007721435_0212 2016-11-02 18:55:04,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0212_01_000001 2016-11-02 18:55:04,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0212_01_000001 for log-aggregation 2016-11-02 18:55:04,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0212 2016-11-02 18:55:05,634 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0212_01_000001] 2016-11-02 18:55:05,634 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0212 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:55:05,635 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0212 2016-11-02 18:55:05,635 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0212 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:55:05,635 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0212 2016-11-02 18:55:05,655 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0212_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:55:56,728 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_e06_1478007721435_0215_01_000004 2016-11-02 18:56:02,858 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0215_000001 (auth:SIMPLE) 2016-11-02 18:56:02,866 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0215_01_000007 by user hdfs 2016-11-02 18:56:02,869 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0215 2016-11-02 18:56:02,870 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0215 transitioned from NEW to INITING 2016-11-02 18:56:02,871 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0215 CONTAINERID=container_e06_1478007721435_0215_01_000007 2016-11-02 18:56:02,871 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0215_01_000007 to application application_1478007721435_0215 2016-11-02 18:56:02,923 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0215 transitioned from INITING to RUNNING 2016-11-02 18:56:02,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0215_01_000007 transitioned from NEW to LOCALIZING 2016-11-02 18:56:02,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0215 2016-11-02 18:56:02,924 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0215_01_000007 2016-11-02 18:56:02,925 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:56:02,925 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0215_01_000007.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:56:02,927 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0215_01_000007 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:56:02,928 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:56:02,928 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:56:02,928 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0215 CONTAINERID=container_e06_1478007721435_0215_01_000007 2016-11-02 18:56:02,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0215_01_000007 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:56:02,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0215_01_000007 from application application_1478007721435_0215 2016-11-02 18:56:02,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0215_01_000007 2016-11-02 18:56:02,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0215_01_000007 for log-aggregation 2016-11-02 18:56:02,929 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0215 2016-11-02 18:56:15,961 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0217_000001 (auth:SIMPLE) 2016-11-02 18:56:15,969 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0217_01_000003 by user hdfs 2016-11-02 18:56:15,973 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0217 2016-11-02 18:56:15,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0217 transitioned from NEW to INITING 2016-11-02 18:56:15,978 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0217 CONTAINERID=container_e06_1478007721435_0217_01_000003 2016-11-02 18:56:15,979 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:56:16,008 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0217_01_000003 to application application_1478007721435_0217 2016-11-02 18:56:16,008 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0217 transitioned from INITING to RUNNING 2016-11-02 18:56:16,009 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0217_01_000003 transitioned from NEW to LOCALIZING 2016-11-02 18:56:16,009 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0217 2016-11-02 18:56:16,009 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0217_01_000003 2016-11-02 18:56:16,009 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:56:16,010 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0217_01_000003.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:56:16,010 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0217_01_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:56:16,010 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 18:56:16,010 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:56:16,011 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0217 CONTAINERID=container_e06_1478007721435_0217_01_000003 2016-11-02 18:56:16,011 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0217_01_000003 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:56:16,011 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0217_01_000003 from application application_1478007721435_0217 2016-11-02 18:56:16,011 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0217_01_000003 2016-11-02 18:56:16,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0217_01_000003 for log-aggregation 2016-11-02 18:56:16,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0217 2016-11-02 18:56:22,021 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0217_01_000003] 2016-11-02 18:56:22,311 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0217_000001 (auth:SIMPLE) 2016-11-02 18:56:22,316 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0217_01_000005 by user hdfs 2016-11-02 18:56:22,318 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0217_01_000005 to application application_1478007721435_0217 2016-11-02 18:56:22,318 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.37.197 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0217 CONTAINERID=container_e06_1478007721435_0217_01_000005 2016-11-02 18:56:22,318 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0217_01_000005 transitioned from NEW to LOCALIZING 2016-11-02 18:56:22,318 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0217 2016-11-02 18:56:22,318 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0217_01_000005 2016-11-02 18:56:22,319 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:56:22,319 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0217_01_000005.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:56:22,319 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0217_01_000005 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:56:22,323 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #1 2016-11-02 18:56:22,323 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #1: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:56:22,323 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0217 CONTAINERID=container_e06_1478007721435_0217_01_000005 2016-11-02 18:56:22,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0217_01_000005 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:56:22,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0217_01_000005 from application application_1478007721435_0217 2016-11-02 18:56:22,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0217_01_000005 2016-11-02 18:56:22,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0217_01_000005 for log-aggregation 2016-11-02 18:56:22,324 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0217 2016-11-02 18:56:46,370 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0219_000001 (auth:SIMPLE) 2016-11-02 18:56:46,379 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0219_01_000001 by user hdfs 2016-11-02 18:56:46,381 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0219 2016-11-02 18:56:46,381 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0219 transitioned from NEW to INITING 2016-11-02 18:56:46,382 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0219_01_000001 to application application_1478007721435_0219 2016-11-02 18:56:46,382 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0219 CONTAINERID=container_e06_1478007721435_0219_01_000001 2016-11-02 18:56:46,435 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0219 transitioned from INITING to RUNNING 2016-11-02 18:56:46,435 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0219_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 18:56:46,435 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0219 2016-11-02 18:56:46,435 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0219_01_000001 2016-11-02 18:56:46,436 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:56:46,436 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0219_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:56:46,436 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0219_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:56:46,437 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #3 2016-11-02 18:56:46,437 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #3: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:56:46,437 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0219 CONTAINERID=container_e06_1478007721435_0219_01_000001 2016-11-02 18:56:46,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0219_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:56:46,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0219_01_000001 from application application_1478007721435_0219 2016-11-02 18:56:46,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0219_01_000001 2016-11-02 18:56:46,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0219_01_000001 for log-aggregation 2016-11-02 18:56:46,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0219 2016-11-02 18:56:47,441 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0219_01_000001] 2016-11-02 18:56:47,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0219 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:56:47,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0219 2016-11-02 18:56:47,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0219 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:56:47,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0219 2016-11-02 18:56:47,461 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0219_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:58:03,562 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0221_000001 (auth:SIMPLE) 2016-11-02 18:58:03,576 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0221_01_000002 by user hdfs 2016-11-02 18:58:03,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0221 2016-11-02 18:58:03,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0221 transitioned from NEW to INITING 2016-11-02 18:58:03,578 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0221_01_000002 to application application_1478007721435_0221 2016-11-02 18:58:03,580 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0221 CONTAINERID=container_e06_1478007721435_0221_01_000002 2016-11-02 18:58:03,586 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Remote Root Log Dir [/tmp/logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-11-02 18:58:03,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0221 transitioned from INITING to RUNNING 2016-11-02 18:58:03,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0221_01_000002 transitioned from NEW to LOCALIZING 2016-11-02 18:58:03,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0221 2016-11-02 18:58:03,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0221_01_000002 2016-11-02 18:58:03,632 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:58:03,633 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0221_01_000002.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 18:58:03,633 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0221_01_000002 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 18:58:03,634 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #2 2016-11-02 18:58:03,634 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #2: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 18:58:03,634 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0221 CONTAINERID=container_e06_1478007721435_0221_01_000002 2016-11-02 18:58:03,637 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0221_01_000002 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 18:58:03,637 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0221_01_000002 from application application_1478007721435_0221 2016-11-02 18:58:03,637 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0221_01_000002 2016-11-02 18:58:03,637 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0221_01_000002 for log-aggregation 2016-11-02 18:58:03,637 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0221 2016-11-02 18:58:04,640 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0221_01_000002] 2016-11-02 18:58:04,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0221 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:58:04,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0221 2016-11-02 18:58:04,645 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0221 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:58:04,645 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0221 2016-11-02 18:58:04,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0221_01_000002. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:58:44,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0166 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:58:44,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0169 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:58:44,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0172 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 18:58:44,743 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0166 2016-11-02 18:58:44,743 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0169 2016-11-02 18:58:44,743 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0172 2016-11-02 18:58:44,744 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0166 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:58:44,744 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0169 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:58:44,744 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0172 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 18:58:44,744 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0166 2016-11-02 18:58:44,744 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0169 2016-11-02 18:58:44,744 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0172 2016-11-02 18:58:44,785 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0166_01_000003. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:58:44,787 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0172_01_000006. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:58:44,792 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0169_01_000002. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 18:58:44,792 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0169_01_000004. Current good log dirs are /var/log/hadoop-yarn/userlogs 2016-11-02 19:00:23,910 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1478007721435_0224_000001 (auth:SIMPLE) 2016-11-02 19:00:23,916 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e06_1478007721435_0224_01_000001 by user hdfs 2016-11-02 19:00:23,920 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478007721435_0224 2016-11-02 19:00:23,920 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0224 transitioned from NEW to INITING 2016-11-02 19:00:23,920 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_e06_1478007721435_0224_01_000001 to application application_1478007721435_0224 2016-11-02 19:00:23,923 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=10.64.36.89 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478007721435_0224 CONTAINERID=container_e06_1478007721435_0224_01_000001 2016-11-02 19:00:23,979 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0224 transitioned from INITING to RUNNING 2016-11-02 19:00:23,979 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0224_01_000001 transitioned from NEW to LOCALIZING 2016-11-02 19:00:23,979 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478007721435_0224 2016-11-02 19:00:23,980 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e06_1478007721435_0224_01_000001 2016-11-02 19:00:23,980 WARN org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not executable: /tmp/hadoop-yarn/nm-local-dir/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:215) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:190) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:124) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 19:00:23,981 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for nmPrivate/container_e06_1478007721435_0224_01_000001.tokens at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:563) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133) 2016-11-02 19:00:23,981 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0224_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2016-11-02 19:00:23,983 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2016-11-02 19:00:23,983 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:269) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:757) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:244) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-02 19:00:23,985 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1478007721435_0224 CONTAINERID=container_e06_1478007721435_0224_01_000001 2016-11-02 19:00:23,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e06_1478007721435_0224_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2016-11-02 19:00:23,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_e06_1478007721435_0224_01_000001 from application application_1478007721435_0224 2016-11-02 19:00:23,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e06_1478007721435_0224_01_000001 2016-11-02 19:00:23,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e06_1478007721435_0224_01_000001 for log-aggregation 2016-11-02 19:00:23,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1478007721435_0224 2016-11-02 19:00:24,989 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e06_1478007721435_0224_01_000001] 2016-11-02 19:00:24,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0224 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-11-02 19:00:24,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478007721435_0224 2016-11-02 19:00:24,991 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1478007721435_0224 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-11-02 19:00:24,991 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478007721435_0224 2016-11-02 19:00:25,009 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e06_1478007721435_0224_01_000001. Current good log dirs are /var/log/hadoop-yarn/userlogs