Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5465

Tasks are often killed before they exit on their own

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished.

      AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing.

      There needs to be better delay before container is killed if profiling is enabled.

      1. MAPREDUCE-5465.patch
        3 kB
        Ravi Prakash
      2. MAPREDUCE-5465-2.patch
        36 kB
        Ming Ma
      3. MAPREDUCE-5465-3.patch
        41 kB
        Ming Ma
      4. MAPREDUCE-5465-4.patch
        89 kB
        Ming Ma
      5. MAPREDUCE-5465-5.patch
        90 kB
        Ming Ma
      6. MAPREDUCE-5465-6.patch
        90 kB
        Ming Ma
      7. MAPREDUCE-5465-7.patch
        86 kB
        Ming Ma
      8. MAPREDUCE-5465-8.patch
        87 kB
        Ming Ma
      9. MAPREDUCE-5465-9.patch
        87 kB
        Ming Ma
      10. MAPREDUCE-5465-branch-2.patch
        86 kB
        Ming Ma

        Issue Links

          Activity

          Hide
          hitesh Hitesh Shah added a comment -

          Is the just a question of setting YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS to a higher value on the cluster that you are testing against?

          Show
          hitesh Hitesh Shah added a comment - Is the just a question of setting YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS to a higher value on the cluster that you are testing against?
          Hide
          hsn Radim Kolar added a comment -

          My original idea is to have 2nd timeout value for use cases where profiling is enabled on particular container.

          Show
          hsn Radim Kolar added a comment - My original idea is to have 2nd timeout value for use cases where profiling is enabled on particular container.
          Hide
          hitesh Hitesh Shah added a comment -

          YARN does not know whether a container is being profiled. Are you suggesting an additional api change in YARN to support the notion of a container being profiled?

          In the scenario mentioned in the description, doesn't it make more sense for the MR application to be changed such that the MR task process goes through a normal shutdown ( after work is finished ) to enable itself to dump out the profiled information? Instead of the MR AM killing the MR task/container via the ContainerManager protocol via a SIGTERM/SIGKILL?

          Show
          hitesh Hitesh Shah added a comment - YARN does not know whether a container is being profiled. Are you suggesting an additional api change in YARN to support the notion of a container being profiled? In the scenario mentioned in the description, doesn't it make more sense for the MR application to be changed such that the MR task process goes through a normal shutdown ( after work is finished ) to enable itself to dump out the profiled information? Instead of the MR AM killing the MR task/container via the ContainerManager protocol via a SIGTERM/SIGKILL?
          Hide
          jlowe Jason Lowe added a comment -

          IMO the AM should always allow the task attempt time to exit successfully on its own rather than sending it a kill signal that races with the normal shutdown of the task attempt. This is very similar to the race between the AM shutting down after unregistering with the RM and the subsequent kill being sent by the RM which was mitigated by MAPREDUCE-4157. This would also help eliminate the many confusing "Container killed by ApplicationMaster" messages that are appearing in task attempt diagnostics for tasks that are otherwise operating normally.

          Show
          jlowe Jason Lowe added a comment - IMO the AM should always allow the task attempt time to exit successfully on its own rather than sending it a kill signal that races with the normal shutdown of the task attempt. This is very similar to the race between the AM shutting down after unregistering with the RM and the subsequent kill being sent by the RM which was mitigated by MAPREDUCE-4157 . This would also help eliminate the many confusing "Container killed by ApplicationMaster" messages that are appearing in task attempt diagnostics for tasks that are otherwise operating normally.
          Hide
          jlowe Jason Lowe added a comment -

          Moved to MAPREDUCE since this is a problem with the MR ApplicationMaster and not YARN.

          Show
          jlowe Jason Lowe added a comment - Moved to MAPREDUCE since this is a problem with the MR ApplicationMaster and not YARN.
          Hide
          raviprak Ravi Prakash added a comment -

          Has any one tried fixing this? If not, I may take a crack it.

          Show
          raviprak Ravi Prakash added a comment - Has any one tried fixing this? If not, I may take a crack it.
          Hide
          raviprak Ravi Prakash added a comment -

          What do you all think of this simple patch?

          Show
          raviprak Ravi Prakash added a comment - What do you all think of this simple patch?
          Hide
          raviprak Ravi Prakash added a comment -

          Actually this patch will probably tie up the dispatcher thread for 250ms for every task. Not a good idea for big jobs. I guess one way might be to queue in a Future to submit the CONTAINER_REMOTE_CLEANUP. Then I'd have to wait for all those events to drain before exit. Considerably more involved

          Show
          raviprak Ravi Prakash added a comment - Actually this patch will probably tie up the dispatcher thread for 250ms for every task. Not a good idea for big jobs. I guess one way might be to queue in a Future to submit the CONTAINER_REMOTE_CLEANUP. Then I'd have to wait for all those events to drain before exit. Considerably more involved
          Hide
          jlowe Jason Lowe added a comment -

          Another way to solve it is to add a separate task state for finishing tasks, similar to the FINISHING state that was added for apps, where the AM is simply waiting for the container completed event or the task to expire.

          Show
          jlowe Jason Lowe added a comment - Another way to solve it is to add a separate task state for finishing tasks, similar to the FINISHING state that was added for apps, where the AM is simply waiting for the container completed event or the task to expire.
          Hide
          raviprak Ravi Prakash added a comment -

          And I totally forgot to answer Hitesh's question! Sorry about that. NM_SLEEP_DELAY_BEFORE_SIGKILL_MS handles the timeout before a SIGKILL. However at that point in time, a SIGTERM has already been sent once. This can be disruptive too.

          Show
          raviprak Ravi Prakash added a comment - And I totally forgot to answer Hitesh's question! Sorry about that. NM_SLEEP_DELAY_BEFORE_SIGKILL_MS handles the timeout before a SIGKILL. However at that point in time, a SIGTERM has already been sent once. This can be disruptive too.
          Hide
          mingma Ming Ma added a comment -

          I discussed with Ravi offline and will provide the patch for review soon.

          The basic approach is to define a new state called FINISHING_CONTAINER for TaskAttemptStateInternal. TaskAttempt will transition to this new state after it receives TaskUmbilicalProtocol's done notification from the task JVM. This will give a chance for the container to exit by itself. Normally the attempt will receive container exit notification via NM -> RM -> AM route; if it doesn't get the notification in time, it will time out and clean up the container via stopContainer.

          Show
          mingma Ming Ma added a comment - I discussed with Ravi offline and will provide the patch for review soon. The basic approach is to define a new state called FINISHING_CONTAINER for TaskAttemptStateInternal. TaskAttempt will transition to this new state after it receives TaskUmbilicalProtocol's done notification from the task JVM. This will give a chance for the container to exit by itself. Normally the attempt will receive container exit notification via NM -> RM -> AM route; if it doesn't get the notification in time, it will time out and clean up the container via stopContainer.
          Hide
          mingma Ming Ma added a comment -

          Here is the patch.

          Show
          mingma Ming Ma added a comment - Here is the patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12632292/MAPREDUCE-5465-2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4481//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12632292/MAPREDUCE-5465-2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4481//console This message is automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Updated patch for the latest trunk.

          Show
          mingma Ming Ma added a comment - Updated patch for the latest trunk.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12638749/MAPREDUCE-5465-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          -1 javac. The applied patch generated 1484 javac compiler warnings (more than the trunk's current 1483 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4486//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4486//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4486//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12638749/MAPREDUCE-5465-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. -1 javac . The applied patch generated 1484 javac compiler warnings (more than the trunk's current 1483 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4486//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4486//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4486//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for picking this up, Ming. Some comments on the patch:

          • The new interface for TaskAttemptListener seems more complicated than necessary. IIUC it's not valid to call registerFinishingTask without having called unregister essentially at the same time, so wondering if we should just add a single method, something like setTaskFinishing(TaskAttemptId, WrappedJvmID) to denote when the task JVM has finished and we're simply waiting for the container to complete. A typical lifecycle could pass a null JVM to the unregister method if the JVM already finished, and we can still skip the finishing method and go straight from registerLauchedTask -> unregister if the task container exits unexpectedly. Either that or it seems the TaskAttemptExitHandler should not be part of TaskAttemptListener and instead conveyed separately to the TaskAttemptImpl (possibly via AppContext along with other things), since it seems tacked on and not related to the other workings of TaskAttemptListener.
          • It looks like we're only handling the successful task case. We don't want to proactively kill tasks that have reported failure just like we don't want to proactively kill tasks that have reported success.
          • Why are we transitioning from FINISHING_CONTAINER to SUCCESS_CONTAINER_CLEANUP rather than to SUCCEEDED when we receive a container completed event? The SUCCESS_CONTAINER_CLEANUP state is for waiting for a container completed event to arrive, but we are leaving the FINISHING_CONTAINER state due to the arrival of that very event.
          • The new properties should be added to mapred-default.xml for documentation purposes.
          • Suggestion: TaskAttemptFinishingMonitor may be a more accurate name than TaskAttemptExitHandler
          • Nit: The default values for the new properties should be named in MRJobConfig
          • Nit: comments should be formatted for 80 columns
          • Nit: In the comment for the new FINISHING_CONTAINER state: "That will a chance" should be "That will give a chance"
          • Nit: testFinshingAttemptTimedout -> testFinishingAttemptTimeout
          • Nit: testTaskAttemptDiognosticEventOnFinishing -> testTaskAttemptDiagnosticEventOnFinishing
          Show
          jlowe Jason Lowe added a comment - Thanks for picking this up, Ming. Some comments on the patch: The new interface for TaskAttemptListener seems more complicated than necessary. IIUC it's not valid to call registerFinishingTask without having called unregister essentially at the same time, so wondering if we should just add a single method, something like setTaskFinishing(TaskAttemptId, WrappedJvmID) to denote when the task JVM has finished and we're simply waiting for the container to complete. A typical lifecycle could pass a null JVM to the unregister method if the JVM already finished, and we can still skip the finishing method and go straight from registerLauchedTask -> unregister if the task container exits unexpectedly. Either that or it seems the TaskAttemptExitHandler should not be part of TaskAttemptListener and instead conveyed separately to the TaskAttemptImpl (possibly via AppContext along with other things), since it seems tacked on and not related to the other workings of TaskAttemptListener. It looks like we're only handling the successful task case. We don't want to proactively kill tasks that have reported failure just like we don't want to proactively kill tasks that have reported success. Why are we transitioning from FINISHING_CONTAINER to SUCCESS_CONTAINER_CLEANUP rather than to SUCCEEDED when we receive a container completed event? The SUCCESS_CONTAINER_CLEANUP state is for waiting for a container completed event to arrive, but we are leaving the FINISHING_CONTAINER state due to the arrival of that very event. The new properties should be added to mapred-default.xml for documentation purposes. Suggestion: TaskAttemptFinishingMonitor may be a more accurate name than TaskAttemptExitHandler Nit: The default values for the new properties should be named in MRJobConfig Nit: comments should be formatted for 80 columns Nit: In the comment for the new FINISHING_CONTAINER state: "That will a chance" should be "That will give a chance" Nit: testFinshingAttemptTimedout -> testFinishingAttemptTimeout Nit: testTaskAttemptDiognosticEventOnFinishing -> testTaskAttemptDiagnosticEventOnFinishing
          Hide
          mingma Ming Ma added a comment -

          Thanks Jason for the review. I will upload the updated patch soon. Want to comment on the couple points you mentioned.

          1. Yes, putting finishTaskMonitor under TaskAttemptListenerImpl isn't clean, given TaskAttemptListenerImpl should only deal with TaskUmbilicalProtocol related. I will move it out to AppContext layer.
          2. Handling of TA_FAILMSG event. TA_FAILMSG can be triggered by task JVM as well as user via "hadoop job -fail-task command". For the case where task JVM reports failure, yes, it can wait for the container to exit. For the case where end users send the command, it will need to clean up the container right away. I skipped that for simplicity. If we want to support that, it seems we will need a new event like TA_FAILMSG_BY_USER.
          3. Why are we transitioning from FINISHING_CONTAINER to SUCCESS_CONTAINER_CLEANUP rather than to SUCCEEDED when we receive a container completed event? It was done for simplicity so that all successful states will go to SUCCESS_CONTAINER_CLEANUP first. But I agree it can go directly to SUCCEEDED when we receive a container completed event.

          Show
          mingma Ming Ma added a comment - Thanks Jason for the review. I will upload the updated patch soon. Want to comment on the couple points you mentioned. 1. Yes, putting finishTaskMonitor under TaskAttemptListenerImpl isn't clean, given TaskAttemptListenerImpl should only deal with TaskUmbilicalProtocol related. I will move it out to AppContext layer. 2. Handling of TA_FAILMSG event. TA_FAILMSG can be triggered by task JVM as well as user via "hadoop job -fail-task command". For the case where task JVM reports failure, yes, it can wait for the container to exit. For the case where end users send the command, it will need to clean up the container right away. I skipped that for simplicity. If we want to support that, it seems we will need a new event like TA_FAILMSG_BY_USER. 3. Why are we transitioning from FINISHING_CONTAINER to SUCCESS_CONTAINER_CLEANUP rather than to SUCCEEDED when we receive a container completed event? It was done for simplicity so that all successful states will go to SUCCESS_CONTAINER_CLEANUP first. But I agree it can go directly to SUCCEEDED when we receive a container completed event.
          Hide
          mingma Ming Ma added a comment -

          Updates per Jason's suggestions.

          1. This patch also includes fix for https://issues.apache.org/jira/browse/MAPREDUCE-5835. Otherwise, some unit tests might fail due to new states introduced.
          2. Fix the handling of TA_CONTAINER_COMPLETED for other cases as well. For example if TA receives TA_CONTAINER_COMPLETED when it is in RUNNING state, it doesn't need to transition to FAIL_CONTAINER_CLEANUP to clean up container.

          Show
          mingma Ming Ma added a comment - Updates per Jason's suggestions. 1. This patch also includes fix for https://issues.apache.org/jira/browse/MAPREDUCE-5835 . Otherwise, some unit tests might fail due to new states introduced. 2. Fix the handling of TA_CONTAINER_COMPLETED for other cases as well. For example if TA receives TA_CONTAINER_COMPLETED when it is in RUNNING state, it doesn't need to transition to FAIL_CONTAINER_CLEANUP to clean up container.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12640939/MAPREDUCE-5465-4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          -1 javac. The applied patch generated 1288 javac compiler warnings (more than the trunk's current 1287 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4539//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4539//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4539//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12640939/MAPREDUCE-5465-4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. -1 javac . The applied patch generated 1288 javac compiler warnings (more than the trunk's current 1287 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4539//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4539//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4539//console This message is automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Updated version that fixes javac and findbug warning.

          Show
          mingma Ming Ma added a comment - Updated version that fixes javac and findbug warning.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641300/MAPREDUCE-5465-5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4545//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4545//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641300/MAPREDUCE-5465-5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4545//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4545//console This message is automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Merge batch with latest trunk.

          Show
          mingma Ming Ma added a comment - Merge batch with latest trunk.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12643435/MAPREDUCE-5465-6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 3 release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.pipes.TestPipeApplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4584//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4584//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4584//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12643435/MAPREDUCE-5465-6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 3 release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4584//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4584//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4584//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          The release audit warnings are unrelated, filed MAPREDUCE-5885. The TestPipeApplication timeout is also unrelated, see MAPREDUCE-5868.

          Thanks for updating the patch, Ming! Sorry for the long delay in getting back to this. I've been thinking about the performance implications of this change. I'm wondering if we should treat the finishing states as if they're the corresponding completed states from external entities (i.e.: task/job). We would send T_ATTEMPT_SUCCEEDED or T_ATTEMPT_FAILED and set task finish times to the time the attempt said it succeeded or failed rather than the time the container completed. Similarly we would map the internal finishing states to their respective external SUCCEEDED/FAILED state rather than RUNNING. From the task/job perspective they're not particularly interested in when the attempt exits, rather they only care about when the task says it's output is available. This would allow the task and job to react to success/failure transitions in the same timeframe that it does today, so there should be a minimal performance impact. The only impact would be if the container needs to complete to free up enough space for the next task's container to be allocated, and in most cases the task will complete quick enough that the AM will receive the new container in the same heartbeat that it used to before this change. Actually this may end up being slightly faster than what it does today, since today it connects to the NM and sends the kill command before it considers the task completed. This proposal would have the task complete as soon as the task indicated via the umbilical.

          Other comments on the latest patch:

          • Rather than have the finishing states call the cleanup container transition and have that transition have to special-case being called by finishing states, it'd be cleaner to factor out the common code from the cleanup container transition that they're trying to leverage and call that instead. Transitions doing state or event checks usually means somethings a bit off, since the transition should already know what event triggered it and what state(s) it applies to.
          • Similarly, the timeout transitions should have dedicated transition code that not only warns in the AM log but also sets an attempt diagnostic message. It can re-use some/all of the cleanup container transition so it's not replicating code. With the diagnostic it will be much more likely the user will be aware of the timeout issue and fix their task code. Tasks that timeout during finishing can still succeed, so users probably won't even know something went wrong unless they bother to examine the AM log and happen to notice it.
          • This change looks like some accidental reformatting:
            --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
            +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
            @@ -222,7 +222,7 @@ public void run() {
                       // remember the current attempt
                       futures.put(event.getTaskAttemptID(), future);
             
            -        } else if (event.getType() == EventType.CONTAINER_REMOTE_CLEANUP) {
            +          } else if (event.getType() == EventType.CONTAINER_REMOTE_CLEANUP) {
             
                       // cancel (and interrupt) the current running task associated with the
                       // event
            
          • Nit: a sendContainerCompleted utility method to send the CONTAINER_COMPLETED event would be nice
          • Nit: code should be formatted to 80 columns, comments for the state transitions in particular.
          Show
          jlowe Jason Lowe added a comment - The release audit warnings are unrelated, filed MAPREDUCE-5885 . The TestPipeApplication timeout is also unrelated, see MAPREDUCE-5868 . Thanks for updating the patch, Ming! Sorry for the long delay in getting back to this. I've been thinking about the performance implications of this change. I'm wondering if we should treat the finishing states as if they're the corresponding completed states from external entities (i.e.: task/job). We would send T_ATTEMPT_SUCCEEDED or T_ATTEMPT_FAILED and set task finish times to the time the attempt said it succeeded or failed rather than the time the container completed. Similarly we would map the internal finishing states to their respective external SUCCEEDED/FAILED state rather than RUNNING. From the task/job perspective they're not particularly interested in when the attempt exits, rather they only care about when the task says it's output is available. This would allow the task and job to react to success/failure transitions in the same timeframe that it does today, so there should be a minimal performance impact. The only impact would be if the container needs to complete to free up enough space for the next task's container to be allocated, and in most cases the task will complete quick enough that the AM will receive the new container in the same heartbeat that it used to before this change. Actually this may end up being slightly faster than what it does today, since today it connects to the NM and sends the kill command before it considers the task completed. This proposal would have the task complete as soon as the task indicated via the umbilical. Other comments on the latest patch: Rather than have the finishing states call the cleanup container transition and have that transition have to special-case being called by finishing states, it'd be cleaner to factor out the common code from the cleanup container transition that they're trying to leverage and call that instead. Transitions doing state or event checks usually means somethings a bit off, since the transition should already know what event triggered it and what state(s) it applies to. Similarly, the timeout transitions should have dedicated transition code that not only warns in the AM log but also sets an attempt diagnostic message. It can re-use some/all of the cleanup container transition so it's not replicating code. With the diagnostic it will be much more likely the user will be aware of the timeout issue and fix their task code. Tasks that timeout during finishing can still succeed, so users probably won't even know something went wrong unless they bother to examine the AM log and happen to notice it. This change looks like some accidental reformatting: --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java @@ -222,7 +222,7 @@ public void run() { // remember the current attempt futures.put(event.getTaskAttemptID(), future); - } else if (event.getType() == EventType.CONTAINER_REMOTE_CLEANUP) { + } else if (event.getType() == EventType.CONTAINER_REMOTE_CLEANUP) { // cancel (and interrupt) the current running task associated with the // event Nit: a sendContainerCompleted utility method to send the CONTAINER_COMPLETED event would be nice Nit: code should be formatted to 80 columns, comments for the state transitions in particular.
          Hide
          mingma Ming Ma added a comment -

          Thanks, Jason! We have discussed the performance implication in https://issues.apache.org/jira/browse/YARN-221. It is good to revisit the issue.

          1. I assume job latency is the metric we want to use. The question is how much such change impacts the job latency.

          2. Say umbilical notification is at t1, task receives T_ATTEMPT_SUCCEEDED or T_ATTEMPT_FAILED at t2, MRAppMaster acquires new containers from RM for next set of tasks at t3.

          3. How much does (t2-t1) impact job latency? It depends on the job characteristics. mapper output can be available sooner; reducer containers can be scheduled sooner, etc. But it isn't going to be linear to number of tasks; given tasks run in parallel. So it should be much smaller. I don't have the formula. It will be useful to compare the performance difference using actual jobs.

          4. Your suggestion of notifying task/job right after t1 is a good idea to improve (t2-t1). I assume it doesn't change the state transition of task attempt. We need to confirm state machine correctness point of view, given there might be some assumptions between task attempt and task state machines.

          5. (t3-t1) can also impact job latency. Notifying task/job earlier won't help to improve (t3-t1).

          6. To improve (t3-t1), perhaps when container exits, it should send OutofBandHeartBeat. Currently OutofBandHeartBeat is sent only when stopContainer is called. Perhaps This is useful when NM->RM's heartbeat interval is big.

          7. It appears there is some issue w.r.t. the current stopContainer's calling NodeStatusUpdaterImpl's OutofBandHeartBeat processing. stopContainer first enqueues "kill" container event before calling NodeStatusUpdaterImpl's OutofBandHeartBeat. So it is possible the NodeStatusUpdaterImpl heartbeat thread sends the heartbeat to RM before the main Dispatcher thread processes the event and mark the container as completed. Thus the OutofBandHeartBeat doesn't include that container in the completed container list. Does it really need to call NodeStatusUpdaterImpl's OutofBandHeartBeat in stopContainer? It seems it is better to call it only when a container exits.

          Show
          mingma Ming Ma added a comment - Thanks, Jason! We have discussed the performance implication in https://issues.apache.org/jira/browse/YARN-221 . It is good to revisit the issue. 1. I assume job latency is the metric we want to use. The question is how much such change impacts the job latency. 2. Say umbilical notification is at t1, task receives T_ATTEMPT_SUCCEEDED or T_ATTEMPT_FAILED at t2, MRAppMaster acquires new containers from RM for next set of tasks at t3. 3. How much does (t2-t1) impact job latency? It depends on the job characteristics. mapper output can be available sooner; reducer containers can be scheduled sooner, etc. But it isn't going to be linear to number of tasks; given tasks run in parallel. So it should be much smaller. I don't have the formula. It will be useful to compare the performance difference using actual jobs. 4. Your suggestion of notifying task/job right after t1 is a good idea to improve (t2-t1). I assume it doesn't change the state transition of task attempt. We need to confirm state machine correctness point of view, given there might be some assumptions between task attempt and task state machines. 5. (t3-t1) can also impact job latency. Notifying task/job earlier won't help to improve (t3-t1). 6. To improve (t3-t1), perhaps when container exits, it should send OutofBandHeartBeat. Currently OutofBandHeartBeat is sent only when stopContainer is called. Perhaps This is useful when NM->RM's heartbeat interval is big. 7. It appears there is some issue w.r.t. the current stopContainer's calling NodeStatusUpdaterImpl's OutofBandHeartBeat processing. stopContainer first enqueues "kill" container event before calling NodeStatusUpdaterImpl's OutofBandHeartBeat. So it is possible the NodeStatusUpdaterImpl heartbeat thread sends the heartbeat to RM before the main Dispatcher thread processes the event and mark the container as completed. Thus the OutofBandHeartBeat doesn't include that container in the completed container list. Does it really need to call NodeStatusUpdaterImpl's OutofBandHeartBeat in stopContainer? It seems it is better to call it only when a container exits.
          Hide
          jlowe Jason Lowe added a comment -

          5. (t3-t1) can also impact job latency. Notifying task/job earlier won't help to improve (t3-t1).

          It can if we're assuming sufficient capacity in the cluster. t3 is dependent upon when the AM asks for the containers, and the sooner the AM knows a task completed the sooner it can ask for new containers (e.g.: map tasks completing leading to launching reduce tasks). The other scenario where job completion time is reduced is when reduce tasks which are already running are waiting upon the final map task. In that case we should be notifying the reduce tasks of the map task completion event as soon as the completion message arrives across the umbilical from the map task and not wait until we receive the container completion from the RM. That delay will directly lead to longer job times.

          Regarding the out-of-band heartbeat, agreed that we should consider sending OOB heartbeats on container completion rather than kill. Filed YARN-2046 to track that issue.

          Show
          jlowe Jason Lowe added a comment - 5. (t3-t1) can also impact job latency. Notifying task/job earlier won't help to improve (t3-t1). It can if we're assuming sufficient capacity in the cluster. t3 is dependent upon when the AM asks for the containers, and the sooner the AM knows a task completed the sooner it can ask for new containers (e.g.: map tasks completing leading to launching reduce tasks). The other scenario where job completion time is reduced is when reduce tasks which are already running are waiting upon the final map task. In that case we should be notifying the reduce tasks of the map task completion event as soon as the completion message arrives across the umbilical from the map task and not wait until we receive the container completion from the RM. That delay will directly lead to longer job times. Regarding the out-of-band heartbeat, agreed that we should consider sending OOB heartbeats on container completion rather than kill. Filed YARN-2046 to track that issue.
          Hide
          mingma Ming Ma added a comment -

          Sorry for the delay.

          Jason, here is the new patch with your suggestion of "to have TA notify task TA_ATTEMPT_SUCCEEDED or TA_ATTEMPT_FAILED after it receives notification from TaskUmbilicalProtocol" and code clean up. Appreciate your input.

          Show
          mingma Ming Ma added a comment - Sorry for the delay. Jason, here is the new patch with your suggestion of "to have TA notify task TA_ATTEMPT_SUCCEEDED or TA_ATTEMPT_FAILED after it receives notification from TaskUmbilicalProtocol" and code clean up. Appreciate your input.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org
          against trunk revision c4c9a78.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4830//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org against trunk revision c4c9a78. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4830//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12664996/MAPREDUCE-5465-7.patch
          against trunk revision c4c9a78.

          -1 @author. The patch appears to contain @author tags which the Hadoop community has agreed to not allow in code contributions.

          +1 tests included. The patch appears to include new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4831//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664996/MAPREDUCE-5465-7.patch against trunk revision c4c9a78. -1 @author . The patch appears to contain @author tags which the Hadoop community has agreed to not allow in code contributions. +1 tests included . The patch appears to include new or modified test files. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4831//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12665193/MAPREDUCE-5465-7.patch
          against trunk revision d8774cc.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          -1 javac. The applied patch generated 1303 javac compiler warnings (more than the trunk's current 1275 warnings).

          -1 javadoc. The javadoc tool appears to have generated 9 warning messages.
          See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4835//artifact/trunk/patchprocess/diffJavadocWarnings.txt for details.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf
          org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs
          org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobConf
          org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts
          org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesTasks
          org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapreduce.security.TestJHSSecurity
          org.apache.hadoop.mapred.TestJobCounters

          The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoopTests
          org.apache.hadoop.mapreducTests
          org.apache.hadoop.mapreTests
          org.apache.haTests
          org.apache.hadoop.maTests
          org.apache.hadoop.mapreduce.v2.appTests
          org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
          org.apache.hadoop.mapreduce.v2.hs.TestHistoryServerFileSystemStateStoreService
          org.apache.hadoop.mapred.TestMultipleLevelCaching
          org.apache.hadoop.mapred.TestReporter
          org.apache.hadoop.mapreduce.TestLargeSort

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4835//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4835//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4835//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665193/MAPREDUCE-5465-7.patch against trunk revision d8774cc. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. -1 javac . The applied patch generated 1303 javac compiler warnings (more than the trunk's current 1275 warnings). -1 javadoc . The javadoc tool appears to have generated 9 warning messages. See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4835//artifact/trunk/patchprocess/diffJavadocWarnings.txt for details. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobConf org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesTasks org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapreduce.security.TestJHSSecurity org.apache.hadoop.mapred.TestJobCounters The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoopTests org.apache.hadoop.mapreducTests org.apache.hadoop.mapreTests org.apache.haTests org.apache.hadoop.maTests org.apache.hadoop.mapreduce.v2.appTests org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler org.apache.hadoop.mapreduce.v2.hs.TestHistoryServerFileSystemStateStoreService org.apache.hadoop.mapred.TestMultipleLevelCaching org.apache.hadoop.mapred.TestReporter org.apache.hadoop.mapreduce.TestLargeSort +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4835//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4835//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4835//console This message is automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Updated patch to address javac warnings.

          Show
          mingma Ming Ma added a comment - Updated patch to address javac warnings.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666142/MAPREDUCE-5465-8.patch
          against trunk revision 08a9ac7.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.app.TestRecovery
          org.apache.hadoop.mapred.TestLineRecordReader
          org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader
          org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo
          org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEntities
          org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter

          The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.TestMRIntermediateDataRunning
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4842//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4842//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666142/MAPREDUCE-5465-8.patch against trunk revision 08a9ac7. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.app.TestRecovery org.apache.hadoop.mapred.TestLineRecordReader org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEntities org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestMRIntermediateDataRunning org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4842//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4842//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666262/MAPREDUCE-5465-8.patch
          against trunk revision ce04621.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4843//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4843//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666262/MAPREDUCE-5465-8.patch against trunk revision ce04621. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4843//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4843//console This message is automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          I'm still catching up on this JIRA for reviewing, but I do have some questions.

          1) I see that SUCCESS_CONTAINER_CLEANUP still exists as a state, but I'm not seeing any transitions to that state. Is it still needed? Or is it a hidden transition now?

          2) I see the instantiation of the ExitFinishingOnTimeoutTransition class, but I'm not seeing it used anywhere. Along similar lines, I'm not seeing any exit transitions from the SUCCESS_FINISHING_CONTAINER and FAIL_FINISHING_CONTAINER classes. I'm also not sure if that is deliberate or not.

          Show
          rchiang Ray Chiang added a comment - I'm still catching up on this JIRA for reviewing, but I do have some questions. 1) I see that SUCCESS_CONTAINER_CLEANUP still exists as a state, but I'm not seeing any transitions to that state. Is it still needed? Or is it a hidden transition now? 2) I see the instantiation of the ExitFinishingOnTimeoutTransition class, but I'm not seeing it used anywhere. Along similar lines, I'm not seeing any exit transitions from the SUCCESS_FINISHING_CONTAINER and FAIL_FINISHING_CONTAINER classes. I'm also not sure if that is deliberate or not.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666262/MAPREDUCE-5465-8.patch
          against trunk revision 61a4c7f.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5314//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666262/MAPREDUCE-5465-8.patch against trunk revision 61a4c7f. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5314//console This message is automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          Ming Ma, Jason Lowe what are your thoughts on the current status of this patch? I'd appreciate any information before I dig too much deeper. Thanks.

          Show
          rchiang Ray Chiang added a comment - Ming Ma , Jason Lowe what are your thoughts on the current status of this patch? I'd appreciate any information before I dig too much deeper. Thanks.
          Hide
          mingma Ming Ma added a comment -

          Ray Chiang, thanks for looking into this.

          SUCCESS_CONTAINER_CLEANUP can be transitioned from SUCCESS_FINISHING_CONTAINER. For ExitFinishingOnTimeoutTransition , you can search for FINISHING_ON_TIMEOUT_TRANSITION.

          We have been running a slight different version of this patch in our production clusters for a while. I can rebase the patch for trunk if people are interested in it.

          Show
          mingma Ming Ma added a comment - Ray Chiang , thanks for looking into this. SUCCESS_CONTAINER_CLEANUP can be transitioned from SUCCESS_FINISHING_CONTAINER. For ExitFinishingOnTimeoutTransition , you can search for FINISHING_ON_TIMEOUT_TRANSITION. We have been running a slight different version of this patch in our production clusters for a while. I can rebase the patch for trunk if people are interested in it.
          Hide
          rchiang Ray Chiang added a comment -

          Thanks for the info.

          And I'd be interested in looking at the latest version that you've got rebased against trunk.

          Show
          rchiang Ray Chiang added a comment - Thanks for the info. And I'd be interested in looking at the latest version that you've got rebased against trunk.
          Hide
          rchiang Ray Chiang added a comment -

          Ming Ma, can you give me an idea when you can get an rebased patch uploaded? Thanks.

          Show
          rchiang Ray Chiang added a comment - Ming Ma , can you give me an idea when you can get an rebased patch uploaded? Thanks.
          Hide
          mingma Ming Ma added a comment -

          Ray, here is the rebased patch.

          Show
          mingma Ming Ma added a comment - Ray, here is the rebased patch.
          Hide
          rchiang Ray Chiang added a comment -

          Great! Thanks!

          Show
          rchiang Ray Chiang added a comment - Great! Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12709076/MAPREDUCE-5465-9.patch
          against trunk revision 6a6a59d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          -1 javac. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.pipes.TestPipeApplication
          org.apache.hadoop.mapred.TestMRTimelineEventHandling
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
          org.apache.hadoop.mapred.TestClusterMRNotification

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5370//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5370//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5370//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12709076/MAPREDUCE-5465-9.patch against trunk revision 6a6a59d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. -1 javac . The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.pipes.TestPipeApplication org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapred.TestClusterMRNotification Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5370//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5370//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5370//console This message is automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          +1 (nonbinding) for version 9 patch.

          • All failed Jenkins unit tests pass in my tree.
          • Version 9 generates a clean state diagram. Version 8 had a state with no transitions to it, while the latest has clean transitions from SUCCESS_FINISHING_CONTAINER to either SUCCESS_CONTAINER_CLEANUP or KILL_CONTAINER_CLEANUP.
          • Running with a modified WordCount job, I get the following results:
            + The non-patched version (running with depth=6) generates a profile.out of 69818 bytes and does not contain the SITES or CPU SAMPLES information.
            + The patched version (running with depth=6) generates a profile.out of 11348778 bytes and contains both the SITES and CPU SAMPLES information.
            + The patched version (running with depth=100) generates a profile.out of 379301524 bytes and contains both the SITES and CPU SAMPLES information. Similar result with depth=10000.
          Show
          rchiang Ray Chiang added a comment - +1 (nonbinding) for version 9 patch. All failed Jenkins unit tests pass in my tree. Version 9 generates a clean state diagram. Version 8 had a state with no transitions to it, while the latest has clean transitions from SUCCESS_FINISHING_CONTAINER to either SUCCESS_CONTAINER_CLEANUP or KILL_CONTAINER_CLEANUP. Running with a modified WordCount job, I get the following results: + The non-patched version (running with depth=6) generates a profile.out of 69818 bytes and does not contain the SITES or CPU SAMPLES information. + The patched version (running with depth=6) generates a profile.out of 11348778 bytes and contains both the SITES and CPU SAMPLES information. + The patched version (running with depth=100) generates a profile.out of 379301524 bytes and contains both the SITES and CPU SAMPLES information. Similar result with depth=10000.
          Hide
          rchiang Ray Chiang added a comment -

          Jason Lowe, I know it's been a while since this has been reviewed by you, but any thoughts? I figured I'd check with you while this was still fresh in my mind.

          Show
          rchiang Ray Chiang added a comment - Jason Lowe , I know it's been a while since this has been reviewed by you, but any thoughts? I figured I'd check with you while this was still fresh in my mind.
          Hide
          jlowe Jason Lowe added a comment -

          Sorry, I have been very busy as of late and haven't had time to get back to this patch. I'll try to get some review comments on the new patch posted by next week.

          Show
          jlowe Jason Lowe added a comment - Sorry, I have been very busy as of late and haven't had time to get back to this patch. I'll try to get some review comments on the new patch posted by next week.
          Hide
          rchiang Ray Chiang added a comment -

          No problem. Thanks for getting back to me.

          Show
          rchiang Ray Chiang added a comment - No problem. Thanks for getting back to me.
          Hide
          rchiang Ray Chiang added a comment -

          Just as a quick follow up, I also did some testing on the mapreduce.task.exit.timeout property. On a machine without extraneous load, it took about 20 seconds to write out the largest profile.out that I could generate. So, having a default timeout of 60 seconds seems like it will be sufficient for all but the most loaded nodes.

          Show
          rchiang Ray Chiang added a comment - Just as a quick follow up, I also did some testing on the mapreduce.task.exit.timeout property. On a machine without extraneous load, it took about 20 seconds to write out the largest profile.out that I could generate. So, having a default timeout of 60 seconds seems like it will be sufficient for all but the most loaded nodes.
          Hide
          jlowe Jason Lowe added a comment -

          Finally carved out some time to take a look at this again. My sincere apologies for the long delay.

          Patch looks great overall, but I noticed that we're ignoring the kill event in the SUCCESS_CONTAINER_CLEANUP. That seems wrong, since we do something significantly different just before or just after that state, in SUCCESS_FINISHING_CONTAINER and SUCCEEDED, respectively. However that issue existed before this patch, so we could address it as a separate JIRA.

          The patch doesn't apply to branch-2. Ming Ma would you mind providing a patch for branch-2 as well?

          Show
          jlowe Jason Lowe added a comment - Finally carved out some time to take a look at this again. My sincere apologies for the long delay. Patch looks great overall, but I noticed that we're ignoring the kill event in the SUCCESS_CONTAINER_CLEANUP. That seems wrong, since we do something significantly different just before or just after that state, in SUCCESS_FINISHING_CONTAINER and SUCCEEDED, respectively. However that issue existed before this patch, so we could address it as a separate JIRA. The patch doesn't apply to branch-2. Ming Ma would you mind providing a patch for branch-2 as well?
          Hide
          mingma Ming Ma added a comment -

          Thanks Jason Lowe for spending time on this. For the handling of TA_KILL event when TA is in SUCCESS_CONTAINER_CLEANUP, do you mean https://issues.apache.org/jira/browse/MAPREDUCE-5776? If so, we can address the issue in that jira. Here is the patch for branch-2.

          Show
          mingma Ming Ma added a comment - Thanks Jason Lowe for spending time on this. For the handling of TA_KILL event when TA is in SUCCESS_CONTAINER_CLEANUP, do you mean https://issues.apache.org/jira/browse/MAPREDUCE-5776? If so, we can address the issue in that jira. Here is the patch for branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732001/MAPREDUCE-5465-branch-2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ea11590
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5704/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732001/MAPREDUCE-5465-branch-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ea11590 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5704/console This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the branch-2 patch, Ming!

          +1 lgtm. Committing this.

          Show
          jlowe Jason Lowe added a comment - Thanks for the branch-2 patch, Ming! +1 lgtm. Committing this.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Ming for the contribution and to Ray for additional review! I committed this to trunk and branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks to Ming for the contribution and to Ray for additional review! I committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7799 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7799/)
          MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7799 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7799/ ) MAPREDUCE-5465 . Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          Hide
          mingma Ming Ma added a comment -

          Thanks Jason and Ray.

          Show
          mingma Ming Ma added a comment - Thanks Jason and Ray.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #925 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/925/)
          MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #925 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/925/ ) MAPREDUCE-5465 . Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/194/)
          MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/194/ ) MAPREDUCE-5465 . Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2141/)
          MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2141/ ) MAPREDUCE-5465 . Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2123/)
          MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2123/ ) MAPREDUCE-5465 . Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/183/)
          MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/183/ ) MAPREDUCE-5465 . Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/193/)
          MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/193/ ) MAPREDUCE-5465 . Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java

            People

            • Assignee:
              mingma Ming Ma
              Reporter:
              hsn Radim Kolar
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development