Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5255

Reduce task preemption results in task never completing , incomplete fix to MAPREDUCE-3858 ?

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.0.3-alpha
    • None
    • mrv2
    • None

    Description

      The problem was seen with symptoms very similar to MAPREDUCE-3858: the job is hung with continuous reduce task attempts, each attempt getting killed around commit phase.

      After a while the single reduce task was the only one remaining in the job, with 50K 'kills' done for the task.

      Relevant logs from application master:

      (the problem task is: attempt_1368653326922_0080_r_001278_0

      2013-05-16 19:27:19,891 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 3
      2013-05-16 19:27:19,892 INFO [IPC Server handler 22 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1368653326922_0080_r_001266_0
      2013-05-16 19:27:19,892 INFO [IPC Server handler 22 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1368653326922_0080_r_001266_0 is : 0.7212161
      2013-05-16 19:27:19,893 INFO [IPC Server handler 13 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1368653326922_0080_r_001266_0
      2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001266_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP
      2013-05-16 19:27:19,893 INFO [ContainerLauncher #19] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1368653326922_0080_01_001296 taskAttempt attempt_1368653326922_0080_r_001266_0
      2013-05-16 19:27:19,893 INFO [ContainerLauncher #19] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1368653326922_0080_r_001266_0
      2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting attempt_1368653326922_0080_r_001279_0
      2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting attempt_1368653326922_0080_r_001278_0
      2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting attempt_1368653326922_0080_r_001277_0
      2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001279_0 TaskAttempt Transitioned from RUNNING to KILL_CONTAINER_CLEANUP
      2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:0 AssignedReds:63 CompletedMaps:16 CompletedReds:1233 ContAlloc:1324 ContRel:25 HostLocal:2 RackLocal:17
      2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_0 TaskAttempt Transitioned from COMMIT_PENDING to KILL_CONTAINER_CLEANUP
      2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001277_0 TaskAttempt Transitioned from RUNNING to KILL_CONTAINER_CLEANUP
      2013-05-16 19:27:19,893 INFO [ContainerLauncher #10] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1368653326922_0080_01_001311 taskAttempt attempt_1368653326922_0080_r_001279_0
      2013-05-16 19:27:19,893 INFO [ContainerLauncher #10] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1368653326922_0080_r_001279_0
      2013-05-16 19:27:19,893 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1368653326922_0080_01_001310 taskAttempt attempt_1368653326922_0080_r_001278_0
      2013-05-16 19:27:19,893 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1368653326922_0080_01_001309 taskAttempt attempt_1368653326922_0080_r_001277_0
      2013-05-16 19:27:19,893 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1368653326922_0080_r_001278_0
      2013-05-16 19:27:19,893 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1368653326922_0080_r_001277_0
      2013-05-16 19:27:19,894 INFO [IPC Server handler 29 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1368653326922_0080_r_001225_0
      2013-05-16 19:27:19,894 INFO [IPC Server handler 29 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1368653326922_0080_r_001225_0 is : 0.7158718
      2013-05-16 19:27:19,894 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001266_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED
      2013-05-16 19:27:19,895 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1368653326922_0080_r_001266_0
      2013-05-16 19:27:19,895 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1368653326922_0080_r_001266 Task Transitioned from RUNNING to SUCCEEDED
      2013-05-16 19:27:19,895 INFO [IPC Server handler 26 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1368653326922_0080_r_001220_0
      2013-05-16 19:27:19,895 INFO [IPC Server handler 24 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1368653326922_0080_r_001225_0
      2013-05-16 19:27:19,895 INFO [IPC Server handler 26 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1368653326922_0080_r_001220_0 is : 0.7038309
      2013-05-16 19:27:19,896 INFO [IPC Server handler 18 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1368653326922_0080_r_001220_0
      2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001279_0 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
      2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1250
      2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001277_0 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
      2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_0 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
      2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001225_0 TaskAtte
      

      Afterwards, we see tasks repeatedly scheduled and killed with the following message:

      e.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:20:54,587 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1368653326922_0080_r_001278_311 : 8080
      2013-05-16 20:20:54,587 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1368653326922_0080_r_001278_311] using containerId: [container_1368653326922_0080_01_001683 on NM: [sjc1-eng-perf-g1-grid01.carrieriq.com:48462]
      2013-05-16 20:20:54,587 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from ASSIGNED to RUNNING
      2013-05-16 20:20:56,320 INFO [IPC Server handler 8 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1368653326922_0080_r_001683 asked for a task
      2013-05-16 20:20:56,321 INFO [IPC Server handler 8 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1368653326922_0080_r_001683 given task: attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:20:57,334 INFO [IPC Server handler 9 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents request from attempt_1368653326922_0080_r_001278_311. startIndex 0 maxEvents 2343
      2013-05-16 20:20:57,549 INFO [IPC Server handler 21 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:20:57,549 INFO [IPC Server handler 21 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1368653326922_0080_r_001278_311 is : 0.0
      2013-05-16 20:20:57,886 INFO [IPC Server handler 15 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:20:57,886 INFO [IPC Server handler 15 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1368653326922_0080_r_001278_311 is : 0.0
      2013-05-16 20:21:00,189 INFO [IPC Server handler 0 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Ping from attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:21:01,766 INFO [IPC Server handler 28 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:21:01,767 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING
      2013-05-16 20:21:01,767 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1368653326922_0080_r_001278_0 already given a go for committing the task output, so killing attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:21:01,767 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from COMMIT_PENDING to KILL_CONTAINER_CLEANUP
      2013-05-16 20:21:01,767 INFO [ContainerLauncher #14] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1368653326922_0080_01_001683 taskAttempt attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:21:01,767 INFO [ContainerLauncher #14] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:21:01,767 INFO [IPC Server handler 20 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1368653326922_0080_r_001278_311
      2013-05-16 20:21:01,767 INFO [IPC Server handler 20 on 40095] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1368653326922_0080_r_001278_311:false
      2013-05-16 20:21:01,769 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
      2013-05-16 20:21:01,769 INFO [TaskCleaner #3] org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl: Processing the event EventType: TASK_CLEAN
      2013-05-16 20:21:01,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
      2013-05-16 20:21:01,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_312 TaskAttempt Transitioned from NEW to UNASSIGNED
      2013-05-16 20:21:02,607 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1 CompletedMaps:19 CompletedReds:1279 ContAlloc:1682 ContRel:67 HostLocal:2 RackLocal:17
      2013-05-16 20:21:02,608 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=1544192
      2013-05-16 20:21:02,608 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:1
      2013-05-16 20:21:02,608 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:1 CompletedMaps:19 CompletedReds:1279 ContAlloc:1682 ContRel:67 HostLocal:2 RackLocal:17
      2013-05-16 20:21:03,611 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1368653326922_0080: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=memory: 1544192 knownNMs=20
      2013-05-16 20:21:03,611 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1368653326922_0080_01_001683
      2013-05-16 20:21:03,611 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0 CompletedMaps:19 CompletedReds:1279 ContAlloc:1682 ContRel:67 HostLocal:2 RackLocal:17
      2013-05-16 20:21:03,611 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1368653326922_0080_r_001278_311: Container killed by the ApplicationMaster.
      

      another one...

      skAttemptImpl: attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
      2013-05-16 20:32:07,857 INFO [ContainerLauncher #17] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1368653326922_0080_01_001753 taskAttempt attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:07,857 INFO [ContainerLauncher #17] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:07,859 INFO [ContainerLauncher #17] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1368653326922_0080_r_001278_377 : 8080
      2013-05-16 20:32:07,860 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1368653326922_0080_r_001278_377] using containerId: [container_1368653326922_0080_01_001753 on NM: [sjc1-eng-perf-g1-grid03.carrieriq.com:58209]
      2013-05-16 20:32:07,860 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from ASSIGNED to RUNNING
      2013-05-16 20:32:09,596 INFO [IPC Server handler 18 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1368653326922_0080_r_001753 asked for a task
      2013-05-16 20:32:09,596 INFO [IPC Server handler 18 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1368653326922_0080_r_001753 given task: attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:10,634 INFO [IPC Server handler 7 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents request from attempt_1368653326922_0080_r_001278_377. startIndex 0 maxEvents 2343
      2013-05-16 20:32:10,871 INFO [IPC Server handler 2 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:10,871 INFO [IPC Server handler 2 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1368653326922_0080_r_001278_377 is : 0.0
      2013-05-16 20:32:11,241 INFO [IPC Server handler 6 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:11,241 INFO [IPC Server handler 6 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1368653326922_0080_r_001278_377 is : 0.0
      2013-05-16 20:32:13,488 INFO [IPC Server handler 12 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Ping from attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:15,085 INFO [IPC Server handler 17 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:15,085 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING
      2013-05-16 20:32:15,085 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1368653326922_0080_r_001278_0 already given a go for committing the task output, so killing attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:15,085 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from COMMIT_PENDING to KILL_CONTAINER_CLEANUP
      2013-05-16 20:32:15,085 INFO [ContainerLauncher #13] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1368653326922_0080_01_001753 taskAttempt attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:15,086 INFO [ContainerLauncher #13] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:15,086 INFO [IPC Server handler 19 on 40095] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1368653326922_0080_r_001278_377
      2013-05-16 20:32:15,086 INFO [IPC Server handler 19 on 40095] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1368653326922_0080_r_001278_377:false
      2013-05-16 20:32:15,088 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
      2013-05-16 20:32:15,088 INFO [TaskCleaner #4] org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl: Processing the event EventType: TASK_CLEAN
      2013-05-16 20:32:15,098 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
      2013-05-16 20:32:15,098 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_378 TaskAttempt Transitioned from NEW to UNASSIGNED
      2013-05-16 20:32:15,878 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1 CompletedMaps:19 CompletedReds:1279 ContAlloc:1752 ContRel:71 HostLocal:2 RackLocal:17
      2013-05-16 20:32:15,879 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=1544192
      2013-05-16 20:32:15,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:1
      2013-05-16 20:32:15,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:1 CompletedMaps:19 CompletedReds:1279 ContAlloc:1752 ContRel:71 HostLocal:2 RackLocal:17
      2013-05-16 20:32:16,895 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1368653326922_0080: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=memory: 1544192 knownNMs=20
      2013-05-16 20:32:16,895 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1368653326922_0080_01_001753
      2013-05-16 20:32:16,895 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0 CompletedMaps:19 CompletedReds:1279 ContAlloc:1752 ContRel:71 HostLocal:2 RackLocal:17
      2013-05-16 20:32:16,895 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1368653326922_0080_r_001278_377: Container killed by the ApplicationMaster.
      
      
      2013-05-16 20:32:17,898 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
      2013-05-16 20:32:17,898 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
      2013-05-16 20:32:17,898 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1368653326922_0080_01_001754 to attempt_1368653326922_0080_r_001278_378
      2013-05-16 20:32:17,898 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1 CompletedMaps:19 CompletedReds:1279 ContAlloc:1753 ContRel:71 HostLocal:2 RackLocal:17
      2013-05-16 20:32:17,898 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sjc1-eng-perf-g1-grid18.carrieriq.com to /default-rack
      2013-05-16 20:32:17,898 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_378 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
      2013-05-16 20:32:17,899 INFO [ContainerLauncher #12] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1368653326922_0080_01_001754 taskAttempt attempt_1368653326922_0080_r_001278_378
      2013-05-16 20:32:17,899 INFO [ContainerLauncher #12] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1368653326922_0080_r_001278_378
      2013-05-16 20:32:17,901 INFO [ContainerLauncher #12] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1368653326922_0080_r_001278_378 : 8080
      2013-05-16 20:32:17,901 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1368653326922_0080_r_001278_378] using containerId: [container_1368653326922_0080_01_001754 on NM: [sjc1-eng-perf-g1-grid18.carrieriq.com:37647]
      2013-05-16 20:32:17,901 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1368653326922_0080_r_001278_378 TaskAttempt Transitioned from ASSIGNED to RUNNING
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            rjain7 Rahul Jain

            Dates

              Created:
              Updated:

              Issue deployment