2014-03-06 19:01:18,587 WARN [main] org.apache.hadoop.conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively 2014-03-06 19:01:19,200 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1394161202967_0004_000001 2014-03-06 19:01:19,679 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-03-06 19:01:19,700 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-03-06 19:01:19,862 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-03-06 19:01:19,885 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2014-03-06 19:01:19,885 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@68daad6) 2014-03-06 19:01:19,913 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: mapreduce.job, Service: job_1394161202967_0003, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@73a3af80) 2014-03-06 19:01:19,914 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 172.16.1.64:52700, Ident: (owner=test, renewer=oozie mr token, realUser=rkanter, issueDate=1394161261984, maxDate=1394766061984, sequenceNumber=9, masterKeyId=2) 2014-03-06 19:01:19,930 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: The specific max attempts: 2 for application: 4. Attempt num: 1 is last retry: false 2014-03-06 19:01:20,108 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-03-06 19:01:20,123 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-03-06 19:01:20,710 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2014-03-06 19:01:20,713 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter 2014-03-06 19:01:20,761 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2014-03-06 19:01:20,762 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2014-03-06 19:01:20,763 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2014-03-06 19:01:20,764 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2014-03-06 19:01:20,765 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2014-03-06 19:01:20,770 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2014-03-06 19:01:20,771 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2014-03-06 19:01:20,772 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2014-03-06 19:01:20,860 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2014-03-06 19:01:21,104 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties 2014-03-06 19:01:21,168 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-03-06 19:01:21,168 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 2014-03-06 19:01:21,178 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1394161202967_0004 to jobTokenSecretManager 2014-03-06 19:01:21,285 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1394161202967_0004 because: not enabled; 2014-03-06 19:01:21,325 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1394161202967_0004 = 12. Number of splits = 2 2014-03-06 19:01:21,327 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1394161202967_0004 = 1 2014-03-06 19:01:21,327 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1394161202967_0004Job Transitioned from NEW to INITED 2014-03-06 19:01:21,329 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1394161202967_0004. 2014-03-06 19:01:21,357 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2014-03-06 19:01:21,365 INFO [Socket Reader #1 for port 52973] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 52973 2014-03-06 19:01:21,386 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 2014-03-06 19:01:21,386 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2014-03-06 19:01:21,387 INFO [IPC Server listener on 52973] org.apache.hadoop.ipc.Server: IPC Server listener on 52973: starting 2014-03-06 19:01:21,387 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at rkanter-mbp.local/172.16.1.64:52973 2014-03-06 19:01:21,470 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2014-03-06 19:01:21,481 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2014-03-06 19:01:21,486 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2014-03-06 19:01:21,486 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2014-03-06 19:01:21,490 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2014-03-06 19:01:21,490 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2014-03-06 19:01:21,507 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 52974 2014-03-06 19:01:22,390 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1394161202967_0004 2014-03-06 19:01:22,392 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2014-03-06 19:01:22,393 INFO [Socket Reader #1 for port 52975] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 52975 2014-03-06 19:01:22,397 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2014-03-06 19:01:22,398 INFO [IPC Server listener on 52975] org.apache.hadoop.ipc.Server: IPC Server listener on 52975: starting 2014-03-06 19:01:22,439 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2014-03-06 19:01:22,439 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2014-03-06 19:01:22,440 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2014-03-06 19:01:22,443 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: hadoop.configured.node.mapping is deprecated. Instead, use net.topology.configured.node.mapping 2014-03-06 19:01:22,487 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-03-06 19:01:22,492 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-03-06 19:01:22,498 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /172.16.1.64:52699 2014-03-06 19:01:22,560 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: 8192 2014-03-06 19:01:22,560 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.test 2014-03-06 19:01:22,563 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2014-03-06 19:01:22,565 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500 2014-03-06 19:01:22,572 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1394161202967_0004Job Transitioned from INITED to SETUP 2014-03-06 19:01:22,574 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 2014-03-06 19:01:22,586 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1394161202967_0004Job Transitioned from SETUP to RUNNING 2014-03-06 19:01:22,607 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved localhost to /default-rack 2014-03-06 19:01:22,613 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1394161202967_0004_m_000000 Task Transitioned from NEW to SCHEDULED 2014-03-06 19:01:22,614 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved localhost to /default-rack 2014-03-06 19:01:22,614 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1394161202967_0004_m_000001 Task Transitioned from NEW to SCHEDULED 2014-03-06 19:01:22,615 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1394161202967_0004_r_000000 Task Transitioned from NEW to SCHEDULED 2014-03-06 19:01:22,616 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2014-03-06 19:01:22,616 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2014-03-06 19:01:22,616 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2014-03-06 19:01:22,618 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceReqt:1024 2014-03-06 19:01:22,624 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceReqt:1024 2014-03-06 19:01:22,646 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1394161202967_0004, File: hdfs://localhost:52681/tmp/hadoop-yarn/staging/test/.staging/job_1394161202967_0004/job_1394161202967_0004_1.jhist 2014-03-06 19:01:23,562 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2014-03-06 19:01:23,614 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit= knownNMs=2 2014-03-06 19:01:23,615 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all scheduled reduces:0 2014-03-06 19:01:23,615 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 0 2014-03-06 19:01:23,616 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=0 2014-03-06 19:01:23,616 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1 2014-03-06 19:01:24,633 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 2 2014-03-06 19:01:24,634 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:24,635 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1394161202967_0004_01_000003 to attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:24,637 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:24,638 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1394161202967_0004_01_000004 to attempt_1394161202967_0004_m_000001_0 2014-03-06 19:01:24,638 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=0 2014-03-06 19:01:24,638 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1 2014-03-06 19:01:24,638 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:2 2014-03-06 19:01:24,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:24,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Job jar is not present. Not adding any jar to the list of resources. 2014-03-06 19:01:24,690 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /tmp/hadoop-yarn/staging/test/.staging/job_1394161202967_0004/job.xml 2014-03-06 19:01:24,698 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #2 tokens and #1 secret keys for NM use for launching container 2014-03-06 19:01:24,698 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 2 2014-03-06 19:01:24,698 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2014-03-06 19:01:24,726 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2014-03-06 19:01:24,728 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:24,729 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2014-03-06 19:01:24,731 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1394161202967_0004_01_000003 taskAttempt attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:24,731 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1394161202967_0004_01_000004 taskAttempt attempt_1394161202967_0004_m_000001_0 2014-03-06 19:01:24,733 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:24,733 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1394161202967_0004_m_000001_0 2014-03-06 19:01:24,734 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:24,734 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : 172.16.1.64:52707 2014-03-06 19:01:24,748 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:24,748 ERROR [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1394161202967_0004_01_000004 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:24,749 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394161202967_0004_m_000001_0: Container launch failed for container_1394161202967_0004_01_000004 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:24,750 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000001_0 TaskAttempt Transitioned from ASSIGNED to FAILED 2014-03-06 19:01:24,750 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE 2014-03-06 19:01:24,756 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved localhost to /default-rack 2014-03-06 19:01:24,757 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node 172.16.1.64 2014-03-06 19:01:24,759 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000001_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2014-03-06 19:01:24,759 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1394161202967_0004_m_000001_1 to list of failed maps 2014-03-06 19:01:24,778 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1394161202967_0004_m_000000_0 : 52709 2014-03-06 19:01:24,779 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1394161202967_0004_m_000000_0] using containerId: [container_1394161202967_0004_01_000003 on NM: [172.16.1.64:52707] 2014-03-06 19:01:24,782 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2014-03-06 19:01:24,782 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: ATTEMPT_START task_1394161202967_0004_m_000000 2014-03-06 19:01:24,782 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1394161202967_0004_m_000000 Task Transitioned from SCHEDULED to RUNNING 2014-03-06 19:01:25,639 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:1 HostLocal:0 RackLocal:2 2014-03-06 19:01:25,673 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=4 release= 1 newContainers=0 finishedContainers=1 resourcelimit= knownNMs=2 2014-03-06 19:01:25,673 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1394161202967_0004_01_000004 2014-03-06 19:01:25,674 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container id container_1394161202967_0004_01_000004 2014-03-06 19:01:25,674 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=0 2014-03-06 19:01:25,674 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1 2014-03-06 19:01:26,681 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2014-03-06 19:01:26,681 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1394161202967_0004_01_000005, NodeId: 172.16.1.64:52707, NodeHttpAddress: 172.16.1.64:52710, Resource: , Priority: 5, Token: Token { kind: ContainerToken, service: 172.16.1.64:52707 }, ] to fast fail map 2014-03-06 19:01:26,681 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 2014-03-06 19:01:26,681 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1394161202967_0004_01_000005 to attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:26,681 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=0 2014-03-06 19:01:26,681 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1 2014-03-06 19:01:26,681 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:1 HostLocal:0 RackLocal:2 2014-03-06 19:01:26,681 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:26,683 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000001_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2014-03-06 19:01:26,684 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1394161202967_0004_01_000005 taskAttempt attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:26,684 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:26,684 INFO [ContainerLauncher #2] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:26,688 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1394161202967_0004_m_000001_1 : 52709 2014-03-06 19:01:26,689 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1394161202967_0004_m_000001_1] using containerId: [container_1394161202967_0004_01_000005 on NM: [172.16.1.64:52707] 2014-03-06 19:01:26,689 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000001_1 TaskAttempt Transitioned from ASSIGNED to RUNNING 2014-03-06 19:01:26,690 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: ATTEMPT_START task_1394161202967_0004_m_000001 2014-03-06 19:01:26,690 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1394161202967_0004_m_000001 Task Transitioned from SCHEDULED to RUNNING 2014-03-06 19:01:27,688 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit= knownNMs=2 2014-03-06 19:01:28,039 INFO [Socket Reader #1 for port 52975] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1394161202967_0004 (auth:SIMPLE) 2014-03-06 19:01:28,068 INFO [IPC Server handler 1 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1394161202967_0004_m_000003 asked for a task 2014-03-06 19:01:28,068 INFO [IPC Server handler 1 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1394161202967_0004_m_000003 given task: attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:29,226 INFO [Socket Reader #1 for port 52975] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1394161202967_0004 (auth:SIMPLE) 2014-03-06 19:01:29,248 INFO [IPC Server handler 0 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1394161202967_0004_m_000005 asked for a task 2014-03-06 19:01:29,248 INFO [IPC Server handler 0 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1394161202967_0004_m_000005 given task: attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:29,844 INFO [IPC Server handler 1 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:29,845 INFO [IPC Server handler 1 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1394161202967_0004_m_000000_0 is : 0.0 2014-03-06 19:01:29,882 INFO [IPC Server handler 0 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:29,882 INFO [IPC Server handler 0 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1394161202967_0004_m_000000_0 is : 1.0 2014-03-06 19:01:29,889 INFO [IPC Server handler 2 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:29,890 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2014-03-06 19:01:29,890 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1394161202967_0004_01_000003 taskAttempt attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:29,891 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:29,891 INFO [ContainerLauncher #3] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:29,900 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000000_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2014-03-06 19:01:29,901 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1394161202967_0004_m_000000_0 2014-03-06 19:01:29,902 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1394161202967_0004_m_000000 Task Transitioned from RUNNING to SUCCEEDED 2014-03-06 19:01:29,903 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1 2014-03-06 19:01:30,617 INFO [IPC Server handler 1 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:30,617 INFO [IPC Server handler 1 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1394161202967_0004_m_000001_1 is : 0.0 2014-03-06 19:01:30,644 INFO [IPC Server handler 0 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:30,644 INFO [IPC Server handler 0 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1394161202967_0004_m_000001_1 is : 1.0 2014-03-06 19:01:30,646 INFO [IPC Server handler 2 on 52975] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:30,647 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000001_1 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2014-03-06 19:01:30,648 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1394161202967_0004_01_000005 taskAttempt attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:30,648 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:30,648 INFO [ContainerLauncher #4] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:30,650 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_m_000001_1 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2014-03-06 19:01:30,651 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1394161202967_0004_m_000001_1 2014-03-06 19:01:30,651 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1394161202967_0004_m_000001 Task Transitioned from RUNNING to SUCCEEDED 2014-03-06 19:01:30,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 2 2014-03-06 19:01:30,698 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:3 ContRel:1 HostLocal:0 RackLocal:2 2014-03-06 19:01:30,727 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1394161202967_0004_01_000003 2014-03-06 19:01:30,727 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=0 2014-03-06 19:01:30,727 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold reached. Scheduling reduces. 2014-03-06 19:01:30,727 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394161202967_0004_m_000000_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2014-03-06 19:01:30,727 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:1 2014-03-06 19:01:30,727 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:1 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:3 ContRel:1 HostLocal:0 RackLocal:2 2014-03-06 19:01:31,731 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit= knownNMs=2 2014-03-06 19:01:31,731 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1394161202967_0004_01_000005 2014-03-06 19:01:31,731 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:3 ContRel:1 HostLocal:0 RackLocal:2 2014-03-06 19:01:31,731 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394161202967_0004_m_000001_1: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2014-03-06 19:01:32,737 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2014-03-06 19:01:32,737 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce 2014-03-06 19:01:32,738 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1394161202967_0004_01_000006 to attempt_1394161202967_0004_r_000000_0 2014-03-06 19:01:32,738 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1 CompletedMaps:2 CompletedReds:0 ContAlloc:4 ContRel:1 HostLocal:0 RackLocal:2 2014-03-06 19:01:32,745 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:32,746 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2014-03-06 19:01:32,746 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1394161202967_0004_01_000006 taskAttempt attempt_1394161202967_0004_r_000000_0 2014-03-06 19:01:32,746 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1394161202967_0004_r_000000_0 2014-03-06 19:01:32,746 INFO [ContainerLauncher #5] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:32,746 ERROR [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1394161202967_0004_01_000006 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:32,747 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394161202967_0004_r_000000_0: Container launch failed for container_1394161202967_0004_01_000006 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:32,747 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_0 TaskAttempt Transitioned from ASSIGNED to FAILED 2014-03-06 19:01:32,747 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE 2014-03-06 19:01:32,748 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node 172.16.1.64 2014-03-06 19:01:32,748 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2014-03-06 19:01:33,738 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:4 ContRel:2 HostLocal:0 RackLocal:2 2014-03-06 19:01:33,742 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=1 release= 1 newContainers=0 finishedContainers=1 resourcelimit= knownNMs=2 2014-03-06 19:01:33,742 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1394161202967_0004_01_000006 2014-03-06 19:01:33,742 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container id container_1394161202967_0004_01_000006 2014-03-06 19:01:33,742 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=0 2014-03-06 19:01:33,742 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:1 2014-03-06 19:01:33,742 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:4 ContRel:2 HostLocal:0 RackLocal:2 2014-03-06 19:01:34,746 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit= knownNMs=2 2014-03-06 19:01:35,751 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2014-03-06 19:01:35,752 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce 2014-03-06 19:01:35,752 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1394161202967_0004_01_000007 to attempt_1394161202967_0004_r_000000_1 2014-03-06 19:01:35,752 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1 CompletedMaps:2 CompletedReds:0 ContAlloc:5 ContRel:2 HostLocal:0 RackLocal:2 2014-03-06 19:01:35,752 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:35,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2014-03-06 19:01:35,754 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1394161202967_0004_01_000007 taskAttempt attempt_1394161202967_0004_r_000000_1 2014-03-06 19:01:35,754 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1394161202967_0004_r_000000_1 2014-03-06 19:01:35,754 INFO [ContainerLauncher #6] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:35,754 ERROR [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1394161202967_0004_01_000007 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:35,755 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394161202967_0004_r_000000_1: Container launch failed for container_1394161202967_0004_01_000007 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:35,755 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_1 TaskAttempt Transitioned from ASSIGNED to FAILED 2014-03-06 19:01:35,755 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE 2014-03-06 19:01:35,756 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node 172.16.1.64 2014-03-06 19:01:35,756 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host 172.16.1.64 2014-03-06 19:01:35,756 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_2 TaskAttempt Transitioned from NEW to UNASSIGNED 2014-03-06 19:01:36,753 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:5 ContRel:3 HostLocal:0 RackLocal:2 2014-03-06 19:01:36,758 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=1 release= 1 newContainers=0 finishedContainers=1 resourcelimit= knownNMs=2 2014-03-06 19:01:36,758 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1394161202967_0004: blacklistAdditions=1 blacklistRemovals=0 2014-03-06 19:01:36,758 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Ignore blacklisting set to true. Known: 2, Blacklisted: 1, 50% 2014-03-06 19:01:36,758 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1394161202967_0004_01_000007 2014-03-06 19:01:36,758 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container id container_1394161202967_0004_01_000007 2014-03-06 19:01:36,759 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=0 2014-03-06 19:01:36,759 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:1 2014-03-06 19:01:36,759 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:5 ContRel:3 HostLocal:0 RackLocal:2 2014-03-06 19:01:37,764 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit= knownNMs=2 2014-03-06 19:01:37,764 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1394161202967_0004: blacklistAdditions=0 blacklistRemovals=1 2014-03-06 19:01:38,770 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2014-03-06 19:01:38,771 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce 2014-03-06 19:01:38,771 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1394161202967_0004_01_000008 to attempt_1394161202967_0004_r_000000_2 2014-03-06 19:01:38,771 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1 CompletedMaps:2 CompletedReds:0 ContAlloc:6 ContRel:3 HostLocal:0 RackLocal:2 2014-03-06 19:01:38,771 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:38,772 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2014-03-06 19:01:38,773 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1394161202967_0004_01_000008 taskAttempt attempt_1394161202967_0004_r_000000_2 2014-03-06 19:01:38,773 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1394161202967_0004_r_000000_2 2014-03-06 19:01:38,773 INFO [ContainerLauncher #7] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:38,774 ERROR [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1394161202967_0004_01_000008 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:38,774 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394161202967_0004_r_000000_2: Container launch failed for container_1394161202967_0004_01_000008 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:38,774 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_2 TaskAttempt Transitioned from ASSIGNED to FAILED 2014-03-06 19:01:38,774 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE 2014-03-06 19:01:38,775 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_3 TaskAttempt Transitioned from NEW to UNASSIGNED 2014-03-06 19:01:39,771 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:6 ContRel:4 HostLocal:0 RackLocal:2 2014-03-06 19:01:39,776 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=1 release= 1 newContainers=0 finishedContainers=1 resourcelimit= knownNMs=2 2014-03-06 19:01:39,776 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1394161202967_0004_01_000008 2014-03-06 19:01:39,776 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container id container_1394161202967_0004_01_000008 2014-03-06 19:01:39,776 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=0 2014-03-06 19:01:39,777 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:1 2014-03-06 19:01:39,777 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0 CompletedMaps:2 CompletedReds:0 ContAlloc:6 ContRel:4 HostLocal:0 RackLocal:2 2014-03-06 19:01:40,780 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1394161202967_0004: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit= knownNMs=2 2014-03-06 19:01:41,792 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2014-03-06 19:01:41,792 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce 2014-03-06 19:01:41,792 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1394161202967_0004_01_000009 to attempt_1394161202967_0004_r_000000_3 2014-03-06 19:01:41,792 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1 CompletedMaps:2 CompletedReds:0 ContAlloc:7 ContRel:4 HostLocal:0 RackLocal:2 2014-03-06 19:01:41,793 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved 172.16.1.64 to /default-rack 2014-03-06 19:01:41,793 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2014-03-06 19:01:41,794 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1394161202967_0004_01_000009 taskAttempt attempt_1394161202967_0004_r_000000_3 2014-03-06 19:01:41,794 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1394161202967_0004_r_000000_3 2014-03-06 19:01:41,794 INFO [ContainerLauncher #8] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: AAA numTokens = 1 NMToken :: 172.16.1.64:52707 :: 172.16.1.64:52707 2014-03-06 19:01:41,795 ERROR [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1394161202967_0004_01_000009 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:41,795 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394161202967_0004_r_000000_3: Container launch failed for container_1394161202967_0004_01_000009 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for 172.16.1.64:52703 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-06 19:01:41,795 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1394161202967_0004_r_000000_3 TaskAttempt Transitioned from ASSIGNED to FAILED 2014-03-06 19:01:41,796 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE 2014-03-06 19:01:41,799 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1394161202967_0004_r_000000 Task Transitioned from SCHEDULED to FAILED 2014-03-06 19:01:41,799 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3 2014-03-06 19:01:41,799 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:0 failedReduces:1 2014-03-06 19:01:41,800 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1394161202967_0004Job Transitioned from RUNNING to FAIL_ABORT 2014-03-06 19:01:41,800 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT 2014-03-06 19:01:41,812 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1394161202967_0004Job Transitioned from FAIL_ABORT to FAILED 2014-03-06 19:01:41,813 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry 2014-03-06 19:01:41,813 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 2014-03-06 19:01:41,813 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true 2014-03-06 19:01:41,813 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 2014-03-06 19:01:41,813 INFO [Thread-63] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 2014-03-06 19:01:41,813 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 2014-03-06 19:01:41,814 INFO [Thread-63] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0 2014-03-06 19:01:41,845 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://localhost:52681/tmp/hadoop-yarn/staging/test/.staging/job_1394161202967_0004/job_1394161202967_0004_1.jhist to hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004-1394161275666-test-oozie%3Aaction%3AT%3Dmap%2Dreduce%3AW%3Dtest%2Dwf%2Dcred%3AA%3Dmap%2Dred-1394161301799-2-0-FAILED-root.test-1394161282567.jhist_tmp 2014-03-06 19:01:41,869 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004-1394161275666-test-oozie%3Aaction%3AT%3Dmap%2Dreduce%3AW%3Dtest%2Dwf%2Dcred%3AA%3Dmap%2Dred-1394161301799-2-0-FAILED-root.test-1394161282567.jhist_tmp 2014-03-06 19:01:41,872 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://localhost:52681/tmp/hadoop-yarn/staging/test/.staging/job_1394161202967_0004/job_1394161202967_0004_1_conf.xml to hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004_conf.xml_tmp 2014-03-06 19:01:41,892 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004_conf.xml_tmp 2014-03-06 19:01:41,900 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004.summary_tmp to hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004.summary 2014-03-06 19:01:41,903 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004_conf.xml_tmp to hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004_conf.xml 2014-03-06 19:01:41,905 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004-1394161275666-test-oozie%3Aaction%3AT%3Dmap%2Dreduce%3AW%3Dtest%2Dwf%2Dcred%3AA%3Dmap%2Dred-1394161301799-2-0-FAILED-root.test-1394161282567.jhist_tmp to hdfs://localhost:52681/tmp/hadoop-yarn/staging/history/done_intermediate/test/job_1394161202967_0004-1394161275666-test-oozie%3Aaction%3AT%3Dmap%2Dreduce%3AW%3Dtest%2Dwf%2Dcred%3AA%3Dmap%2Dred-1394161301799-2-0-FAILED-root.test-1394161282567.jhist 2014-03-06 19:01:41,905 INFO [Thread-63] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2014-03-06 19:01:41,907 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Task failed task_1394161202967_0004_r_000000 Job failed as tasks failed. failedMaps:0 failedReduces:1 2014-03-06 19:01:41,908 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://172.16.1.64:52696/jobhistory/job/job_1394161202967_0004 2014-03-06 19:01:41,914 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered. 2014-03-06 19:01:42,916 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1 CompletedMaps:2 CompletedReds:0 ContAlloc:7 ContRel:4 HostLocal:0 RackLocal:2 2014-03-06 19:01:42,918 INFO [Thread-63] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://localhost:52681 /tmp/hadoop-yarn/staging/test/.staging/job_1394161202967_0004 2014-03-06 19:01:42,922 INFO [Thread-63] org.apache.hadoop.ipc.Server: Stopping server on 52975 2014-03-06 19:01:42,924 INFO [IPC Server listener on 52975] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 52975 2014-03-06 19:01:42,924 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2014-03-06 19:01:42,924 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted