Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4128

AM Recovery expects all attempts of a completed task to also be completed.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:

      Description

      The AM seems to assume that all attempts of a completed task (from a previous AM incarnation) would also be completed. There is at least one case in which this does not hold. Case being cancellation of a completed task resulting in a new running attempt.

      1. MAPREDUCE-4128.patch
        17 kB
        Bikas Saha
      2. MAPREDUCE-4128-1.patch
        17 kB
        Bikas Saha
      3. MAPREDUCE-4128-2.patch
        18 kB
        Bikas Saha

        Issue Links

          Activity

          Hide
          Bikas Saha added a comment -

          In recovery mode, the recovery service assigns a success status to any task that has a FINISHED event reported. The only way that status could be changed is if there is a FAILED event for that task, in which case a failed status is assigned to that task. So once a task is marked with a success status, it remains so even when subsequent events kill the successful task attempt and marks it invalid.
          Next the recovery service adds all success status tasks into a completedTasks collection. Then it proceeds to enumerate the events and process them. When it hits a TaskEventType.*_KILLED/FAILED/SUCCEEDED then it removes those attempts from the completedTasks. Recovery does not complete until all attempts of all completedTasks are removed. Now the following sequence of events can happen for Tasks A and B. A1 represents task attempt 1 of A.
          CompletedTasks contains A and B. A1 and A2 are succeeded. A2 was a rerun of A1. B1 is succeeded and B2 was running when AM crashed.
          A1- container request is processed. It uses the nodeid info from A1 to work.
          B1- container request is processed. It uses the nodeid info from B1 to work.
          A1- Succeeded. removes A1
          B1- Succeeded. removes B1
          A2- container request is processed. It uses the nodeid info from A2 to work
          B2- container request is processed. It uses the nodeid info from B2 to work. But there is no such info as it is populated on task attempt completion. AM crashed here while trying to resolve the nodeid.
          If AM had not crashed the following would have happened
          A2- Succeeded. removes A2
          There is no FAILED/KILLED/SUCCEEDED event for B2 since it was running when the AM crashed. So it seems the AM would never move out of recovery.

          If the above is correct, there seems to be 2 problems
          1) While recovery is in process, event handling for task attempts that are not in a completed state. I am not sure if the recovery design allows this and the current crash is simply a case of missing info.
          2) Expecting every task attempt of a completedTask to have a KILLED/FAILED/SUCCEEDED entry. This seems to be clearly wrong in the current scenario.

          Show
          Bikas Saha added a comment - In recovery mode, the recovery service assigns a success status to any task that has a FINISHED event reported. The only way that status could be changed is if there is a FAILED event for that task, in which case a failed status is assigned to that task. So once a task is marked with a success status, it remains so even when subsequent events kill the successful task attempt and marks it invalid. Next the recovery service adds all success status tasks into a completedTasks collection. Then it proceeds to enumerate the events and process them. When it hits a TaskEventType.*_KILLED/FAILED/SUCCEEDED then it removes those attempts from the completedTasks. Recovery does not complete until all attempts of all completedTasks are removed. Now the following sequence of events can happen for Tasks A and B. A1 represents task attempt 1 of A. CompletedTasks contains A and B. A1 and A2 are succeeded. A2 was a rerun of A1. B1 is succeeded and B2 was running when AM crashed. A1- container request is processed. It uses the nodeid info from A1 to work. B1- container request is processed. It uses the nodeid info from B1 to work. A1- Succeeded. removes A1 B1- Succeeded. removes B1 A2- container request is processed. It uses the nodeid info from A2 to work B2- container request is processed. It uses the nodeid info from B2 to work. But there is no such info as it is populated on task attempt completion. AM crashed here while trying to resolve the nodeid. If AM had not crashed the following would have happened A2- Succeeded. removes A2 There is no FAILED/KILLED/SUCCEEDED event for B2 since it was running when the AM crashed. So it seems the AM would never move out of recovery. If the above is correct, there seems to be 2 problems 1) While recovery is in process, event handling for task attempts that are not in a completed state. I am not sure if the recovery design allows this and the current crash is simply a case of missing info. 2) Expecting every task attempt of a completedTask to have a KILLED/FAILED/SUCCEEDED entry. This seems to be clearly wrong in the current scenario.
          Hide
          Bikas Saha added a comment -

          I have managed to repro this using a modified TestFetchFailure. So this is unrelated to the specific changes in MAPREDUCE-3921.

          Show
          Bikas Saha added a comment - I have managed to repro this using a modified TestFetchFailure. So this is unrelated to the specific changes in MAPREDUCE-3921 .
          Hide
          Bikas Saha added a comment -

          The current recovery mechanism seems to be designed to recover completed tasks. Hence it assumes that all attempts of such tasks would also be complete. So it loads completed tasks and replays them till the replay completes all attempts of all completed tasks.
          This will break whenever there is an attempt running after a previously successful completion because the replay will not have info to correctly work on running attempts.
          Scenario 1 : MAPREDUCE-3921 introduces such an instance because it re-runs successful map tasks if the successful attempts had run on a bad machine.
          Scenario 2 : Even in the current code, when a successful map is rerun because of too many fetch failures, the above scenario is produced and caused a failure in recovery.
          The proposed solution in the patch is to make sure that if a task is re-run then it is not marked as completed during recovery. The JobHistoryParser has been changed to remove the "SUCCEEDED" status on a task if the successful attempt of that task later reports a failure. This fixes the repro case mentioned above. I have improved that testcase to cover Scenario 1. Scenario 2 will be covered in MAPREDUCE-3921.
          I am expecting the patch to introduce 3 additional warnings because of raw types in event handling (similar to existing warnings).
          I compiled a broken rumen class and test assuming the new field added to TaskFinishedEvent is not relevant to them.

          Show
          Bikas Saha added a comment - The current recovery mechanism seems to be designed to recover completed tasks. Hence it assumes that all attempts of such tasks would also be complete. So it loads completed tasks and replays them till the replay completes all attempts of all completed tasks. This will break whenever there is an attempt running after a previously successful completion because the replay will not have info to correctly work on running attempts. Scenario 1 : MAPREDUCE-3921 introduces such an instance because it re-runs successful map tasks if the successful attempts had run on a bad machine. Scenario 2 : Even in the current code, when a successful map is rerun because of too many fetch failures, the above scenario is produced and caused a failure in recovery. The proposed solution in the patch is to make sure that if a task is re-run then it is not marked as completed during recovery. The JobHistoryParser has been changed to remove the "SUCCEEDED" status on a task if the successful attempt of that task later reports a failure. This fixes the repro case mentioned above. I have improved that testcase to cover Scenario 1. Scenario 2 will be covered in MAPREDUCE-3921 . I am expecting the patch to introduce 3 additional warnings because of raw types in event handling (similar to existing warnings). I compiled a broken rumen class and test assuming the new field added to TaskFinishedEvent is not relevant to them.
          Hide
          Bikas Saha added a comment -

          Correction.

          This fixes the repro case mentioned above. I have improved that testcase to cover Scenario 1 Scenario 2. Scenario 2 Scenario 1 will be covered in MAPREDUCE-3921.

          Show
          Bikas Saha added a comment - Correction. This fixes the repro case mentioned above. I have improved that testcase to cover Scenario 1 Scenario 2. Scenario 2 Scenario 1 will be covered in MAPREDUCE-3921 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522049/MAPREDUCE-4128.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2180//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522049/MAPREDUCE-4128.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2180//console This message is automatically generated.
          Hide
          Bikas Saha added a comment -

          patch applies fine on another clean repo on my machine.

          Show
          Bikas Saha added a comment - patch applies fine on another clean repo on my machine.
          Hide
          Robert Joseph Evans added a comment -

          The reason you patch does not apply is because you made a change to hadoop-tools, which the pre-commit build chocks on.

          One issue I saw was TaskFinishedEvent constructor added in a new attemptId, but no javadocs for that param. It is minor but would be a -1 in the Precommit-build.

          I will run the precommit build myself and post the results.

          Show
          Robert Joseph Evans added a comment - The reason you patch does not apply is because you made a change to hadoop-tools, which the pre-commit build chocks on. One issue I saw was TaskFinishedEvent constructor added in a new attemptId, but no javadocs for that param. It is minor but would be a -1 in the Precommit-build. I will run the precommit build myself and post the results.
          Hide
          Bikas Saha added a comment -

          Fixed the javadoc. Thanks!

          Show
          Bikas Saha added a comment - Fixed the javadoc. Thanks!
          Hide
          Robert Joseph Evans added a comment -

          I ran the tests manually, but I ran them against all of Hadoop, not just mapreduce-project so the counts for the -1s are likely to be off.

          -1 overall.  Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12522297/MAPREDUCE-4128-1.patch
            against trunk revision .
          
              +1 @author.  The patch does not contain any @author tags.
          
              +1 tests included.  The patch appears to include 2 new or modified test files.
          
              -1 javadoc.  The javadoc tool appears to have generated 14 warning messages.
          
              -1 javac.  The applied patch generated 1731 javac compiler warnings (more than the trunk's current 1728 warnings).
          
              +1 eclipse:eclipse.  The patch built with eclipse:eclipse.
          
              -1 findbugs.  The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings.
          
              +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
              -1 core tests.  The patch failed these unit tests:
                            org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat
                            org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
                            org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree
                            org.apache.hadoop.mapreduce.v2.TestMRJobs
                            org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEntities
                            org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures
                            org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes
          
              +1 contrib tests.  The patch passed contrib unit tests.
          

          The javadoc warnings look like they can be ignored.
          The javac warnings need to be addressed though. Here is a diff between trunk, and the new patch (yes they are copy and paste but we need to put in the @SurpressWarnings or something.

          1279,1282c1279,1285
          < [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[66,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          < [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[107,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          < [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[115,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          < [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[148,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          ---
          > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[69,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[110,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[118,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[179,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[232,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[241,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[252,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler
          

          The findbugs warnings also look OK. Nothing new, just the root config is off.

          You may want to check the mr/yarn unit tests to be sure they are known issues.

          Show
          Robert Joseph Evans added a comment - I ran the tests manually, but I ran them against all of Hadoop, not just mapreduce-project so the counts for the -1s are likely to be off. -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522297/MAPREDUCE-4128-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 14 warning messages. -1 javac. The applied patch generated 1731 javac compiler warnings (more than the trunk's current 1728 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEntities org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes +1 contrib tests. The patch passed contrib unit tests. The javadoc warnings look like they can be ignored. The javac warnings need to be addressed though. Here is a diff between trunk, and the new patch (yes they are copy and paste but we need to put in the @SurpressWarnings or something. 1279,1282c1279,1285 < [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[66,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler < [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[107,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler < [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[115,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler < [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[148,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler --- > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[69,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[110,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[118,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[179,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[232,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[241,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler > [WARNING] /home/evans/src/commit/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java:[252,45] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler The findbugs warnings also look OK. Nothing new, just the root config is off. You may want to check the mr/yarn unit tests to be sure they are known issues.
          Hide
          Bikas Saha added a comment -

          I am rerunning the tests. Does not look like any of them have to do with recovery though.

          I would prefer leaving the javac warnings there. If/When we figure out how to fix these then we dont want to lose any of them because some got hidden with @suppress. If we do decide suppress them then we could suppress them all in 1 jira for tracking. Sounds good?

          Show
          Bikas Saha added a comment - I am rerunning the tests. Does not look like any of them have to do with recovery though. I would prefer leaving the javac warnings there. If/When we figure out how to fix these then we dont want to lose any of them because some got hidden with @suppress. If we do decide suppress them then we could suppress them all in 1 jira for tracking. Sounds good?
          Hide
          Bikas Saha added a comment -

          All tests pass except TestJobHistoryEntities. TestJobHistoryEntities tries to load a pre-generated history file. Now when Avro tries to parse that file and populate the TaskFinishedEvent it tries to assign a long value at index 2 (index 2 now has a string value) because of the addition of successful task attempt id string.

          Was I wrong in inserting the new field in the middle of the record instead of the end. shouldnt avro be handling this for me?

          Show
          Bikas Saha added a comment - All tests pass except TestJobHistoryEntities. TestJobHistoryEntities tries to load a pre-generated history file. Now when Avro tries to parse that file and populate the TaskFinishedEvent it tries to assign a long value at index 2 (index 2 now has a string value) because of the addition of successful task attempt id string. Was I wrong in inserting the new field in the middle of the record instead of the end. shouldnt avro be handling this for me?
          Hide
          Robert Joseph Evans added a comment -

          We have over 1700 javac warnings right now. I don't think 3 more of the same type are a big deal. But I really don't see anyone ever going back to clean them up, no matter how much I would like to have it happen.

          Show
          Robert Joseph Evans added a comment - We have over 1700 javac warnings right now. I don't think 3 more of the same type are a big deal. But I really don't see anyone ever going back to clean them up, no matter how much I would like to have it happen.
          Hide
          Bikas Saha added a comment -

          yeah. I asked the same question in my previous reviews but there was no clear clean method to solve this.
          I guess we might have to add a global suppress for this specific warning. Having tons of @suppress on all these functions would be messy.

          Do you have any clues on the Avro thing? My web search did not return anything useful. I moved the new successfulTaskAttempt field to the end. That led to NPE later on because Avro set null values into my TaskFinishedEvent.successfulAttemptId. So I had to check for null in TaskFinishedEvent. I dont like it this way.

          Now the test passes. Is there a better way to do this?

          Show
          Bikas Saha added a comment - yeah. I asked the same question in my previous reviews but there was no clear clean method to solve this. I guess we might have to add a global suppress for this specific warning. Having tons of @suppress on all these functions would be messy. Do you have any clues on the Avro thing? My web search did not return anything useful. I moved the new successfulTaskAttempt field to the end. That led to NPE later on because Avro set null values into my TaskFinishedEvent.successfulAttemptId. So I had to check for null in TaskFinishedEvent. I dont like it this way. Now the test passes. Is there a better way to do this?
          Hide
          Bikas Saha added a comment -

          yeah. I asked the same question in my previous reviews but there was no clear clean method to solve this.
          I guess we might have to add a global suppress for this specific warning. Having tons of @suppress on all these functions would be messy.

          Do you have any clues on the Avro thing? My web search did not return anything useful. I moved the new successfulTaskAttempt field to the end. That led to NPE later on because Avro set null values into my TaskFinishedEvent.successfulAttemptId. So I had to check for null in TaskFinishedEvent. I dont like it this way.

          Now the test passes. Is there a better way to do this?

          Show
          Bikas Saha added a comment - yeah. I asked the same question in my previous reviews but there was no clear clean method to solve this. I guess we might have to add a global suppress for this specific warning. Having tons of @suppress on all these functions would be messy. Do you have any clues on the Avro thing? My web search did not return anything useful. I moved the new successfulTaskAttempt field to the end. That led to NPE later on because Avro set null values into my TaskFinishedEvent.successfulAttemptId. So I had to check for null in TaskFinishedEvent. I dont like it this way. Now the test passes. Is there a better way to do this?
          Hide
          Bikas Saha added a comment -

          Attaching patch with above mentioned fix.

          Show
          Bikas Saha added a comment - Attaching patch with above mentioned fix.
          Hide
          Robert Joseph Evans added a comment -

          I am not an expert on Avro, so I really don't know why it is not handling the schema thing better. The JobHistory files are also AVRO JSON, so I am really confused why it would blow up so badly. In either case I think you do need to handle the null for the ID. That is because there are likely to be older job history files sitting around, and we don't want to crash if they are read in.

          Show
          Robert Joseph Evans added a comment - I am not an expert on Avro, so I really don't know why it is not handling the schema thing better. The JobHistory files are also AVRO JSON, so I am really confused why it would blow up so badly. In either case I think you do need to handle the null for the ID. That is because there are likely to be older job history files sitting around, and we don't want to crash if they are read in.
          Hide
          Bikas Saha added a comment -

          This test seems to be for the jobhistory files that you mention.
          I have added the null handling in the latest patch.

          Show
          Bikas Saha added a comment - This test seems to be for the jobhistory files that you mention. I have added the null handling in the latest patch.
          Hide
          Robert Joseph Evans added a comment -

          I looked at the patch and ran a few manual tests and it looks good to me +1. I don't have time to check it in right now, but I will tomorrow if no one else has done it before then. Thanks for your hard work Bikas.

          Show
          Robert Joseph Evans added a comment - I looked at the patch and ran a few manual tests and it looks good to me +1. I don't have time to check it in right now, but I will tomorrow if no one else has done it before then. Thanks for your hard work Bikas.
          Hide
          Bikas Saha added a comment -

          Thanks! That will finally unblock MAPREDUCE-3921.

          Show
          Bikas Saha added a comment - Thanks! That will finally unblock MAPREDUCE-3921 .
          Hide
          Robert Joseph Evans added a comment -

          Thanks for your work on this Bikas. I have put this into trunk, branch-2 and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thanks for your work on this Bikas. I have put this into trunk, branch-2 and branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2143 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2143/)
          MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2143 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2143/ ) MAPREDUCE-4128 . AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2070 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2070/)
          MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2070 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2070/ ) MAPREDUCE-4128 . AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2084 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2084/)
          MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2084 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2084/ ) MAPREDUCE-4128 . AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #227 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/227/)
          svn merge -c 1325765 from trunk. FIXES: MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325768)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325768
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
          • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #227 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/227/ ) svn merge -c 1325765 from trunk. FIXES: MAPREDUCE-4128 . AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325768) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325768 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1014 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1014/)
          MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1014 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1014/ ) MAPREDUCE-4128 . AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1049 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1049/)
          MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1049 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1049/ ) MAPREDUCE-4128 . AM Recovery expects all attempts of a completed task to also be completed. (Bikas Saha via bobby) (Revision 1325765) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325765 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Task20LineHistoryEventEmitter.java

            People

            • Assignee:
              Bikas Saha
              Reporter:
              Bikas Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development