Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6892

Issues with the count of failed/killed tasks in the jhist file

    Details

    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      This adds some new job counters, the number of failed MAP/REDUCE tasks and the number of killed MAP/REDUCE tasks.

      Description

      Recently we encountered some issues with the value of failed tasks. After parsing the jhist file, JobInfo.getFailedMaps() returned 0, but actually there were failures.

      Another minor thing is that you cannot get the number of killed tasks (although this can be calculated).

      The root cause is that JobUnsuccessfulCompletionEvent contains only the successful map/reduce task counts. Number of failed (or killed) tasks are not stored.

      1. MAPREDUCE-6892-001.patch
        30 kB
        Peter Bacsko
      2. MAPREDUCE-6892-002.PATCH
        38 kB
        Peter Bacsko
      3. MAPREDUCE-6892-003.patch
        67 kB
        Peter Bacsko
      4. MAPREDUCE-6892-004.patch
        68 kB
        Peter Bacsko
      5. MAPREDUCE-6892-005.patch
        70 kB
        Peter Bacsko
      6. MAPREDUCE-6892-006.patch
        70 kB
        Peter Bacsko

        Activity

        Hide
        pbacsko Peter Bacsko added a comment - - edited

        Approach to the solution:

        1. Update the Avro schema of JobUnsuccessfulCompletion and JobFinished
        2. Update JobFinishedEvent with the number of killed maps/reduces
        3. Modify JobImpl to set the number of killed maps/reduces in JobFinishedEvent (inside createJobFinishedEvent())
        4. Update JobUnsuccessfulCompletionEvent so that it contains the number of failed/killed maps and reduces
        5. Modify JobHistoryEventHandler to set the new fields in JobUnsuccessfulCompletionEvent

        In order to do #5, we have to know the number of failed/killed attempts which is currently cannot be retrieved from the Job interface. So we either add these methods (and return killedMapTaskCount, killedReduceTaskCount etc in JobImpl) or we retrieve all tasks with getTasks(), then check all task attempts individually and counting the number of killed/failed attempts.

        Modifying org.apache.hadoop.mapreduce.v2.app.job.Job might break things outside Hadoop - I don't know if this is something that we can do without announcing (there's no @InterfaceAudience annotation on it).

        Storing the number of killed task is not an absolute must, but IMO it's good to have convenience methods to retrieve it.

        Show
        pbacsko Peter Bacsko added a comment - - edited Approach to the solution: 1. Update the Avro schema of JobUnsuccessfulCompletion and JobFinished 2. Update JobFinishedEvent with the number of killed maps/reduces 3. Modify JobImpl to set the number of killed maps/reduces in JobFinishedEvent (inside createJobFinishedEvent() ) 4. Update JobUnsuccessfulCompletionEvent so that it contains the number of failed/killed maps and reduces 5. Modify JobHistoryEventHandler to set the new fields in JobUnsuccessfulCompletionEvent In order to do #5, we have to know the number of failed/killed attempts which is currently cannot be retrieved from the Job interface. So we either add these methods (and return killedMapTaskCount , killedReduceTaskCount etc in JobImpl ) or we retrieve all tasks with getTasks() , then check all task attempts individually and counting the number of killed/failed attempts. Modifying org.apache.hadoop.mapreduce.v2.app.job.Job might break things outside Hadoop - I don't know if this is something that we can do without announcing (there's no @InterfaceAudience annotation on it). Storing the number of killed task is not an absolute must, but IMO it's good to have convenience methods to retrieve it.
        Hide
        pbacsko Peter Bacsko added a comment - - edited

        I uploaded first version of the patch. It's more like a POC at this point, I haven't added/modified any tests (I ran testHistoryParsingForFailedAttempts and the parsed JobInfo seems to be correct).

        Show
        pbacsko Peter Bacsko added a comment - - edited I uploaded first version of the patch. It's more like a POC at this point, I haven't added/modified any tests (I ran testHistoryParsingForFailedAttempts and the parsed JobInfo seems to be correct).
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        0 mvndep 1m 34s Maven dependency ordering for branch
        +1 mvninstall 12m 52s trunk passed
        +1 compile 7m 14s trunk passed
        +1 checkstyle 1m 35s trunk passed
        +1 mvnsite 1m 27s trunk passed
        +1 mvneclipse 0m 58s trunk passed
        +1 findbugs 2m 15s trunk passed
        +1 javadoc 1m 7s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 12s the patch passed
        +1 compile 7m 11s the patch passed
        +1 javac 7m 11s the patch passed
        +1 checkstyle 1m 36s the patch passed
        +1 mvnsite 1m 27s the patch passed
        +1 mvneclipse 0m 56s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 42s the patch passed
        -1 javadoc 0m 18s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 58 new + 2496 unchanged - 0 fixed = 2554 total (was 2496)
        +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed.
        +1 unit 8m 36s hadoop-mapreduce-client-app in the patch passed.
        +1 unit 2m 55s hadoop-mapreduce-client-hs in the patch passed.
        +1 unit 0m 18s hadoop-rumen in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        61m 28s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869863/MAPREDUCE-6892-001.patch
        JIRA Issue MAPREDUCE-6892
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4c1dcadb3533 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1a56a3d
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6985/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6985/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6985/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 1m 34s Maven dependency ordering for branch +1 mvninstall 12m 52s trunk passed +1 compile 7m 14s trunk passed +1 checkstyle 1m 35s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 58s trunk passed +1 findbugs 2m 15s trunk passed +1 javadoc 1m 7s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 7m 11s the patch passed +1 javac 7m 11s the patch passed +1 checkstyle 1m 36s the patch passed +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 56s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 42s the patch passed -1 javadoc 0m 18s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 58 new + 2496 unchanged - 0 fixed = 2554 total (was 2496) +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 36s hadoop-mapreduce-client-app in the patch passed. +1 unit 2m 55s hadoop-mapreduce-client-hs in the patch passed. +1 unit 0m 18s hadoop-rumen in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 61m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869863/MAPREDUCE-6892-001.patch JIRA Issue MAPREDUCE-6892 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4c1dcadb3533 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1a56a3d Default Java 1.8.0_131 findbugs v3.1.0-RC1 javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6985/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6985/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6985/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        haibochen Haibo Chen added a comment -

        Thanks Peter Backlund for reporting and fixing the issue. The map/reduce counters in JobUnsuccessfulCompletionEvent are inconsistent. In some cases, # of successful mappers is passed to JobUnsuccessfulCompletion.finishedMaps. In other cases, # of successful + failed + killed mappers is reported. I think we should make it as least consistent, and then add failed/killed mapper/reducer count.

        org.apache.hadoop.mapreduce.v2.app.job.Job is not client facing as far as I can tell, so I think it's fine to add a few more methods (killedMapTaskCount, killedReduceTaskCount). In upgrade cases though, JHS expects the new JobUnsuccessfulCompletion and JobFinishedEvent schema, but it could pick up old .jhist file sthat do not conform with the new schema, we want to make sure it handles the special situation gracefully. Can you check that?

        Show
        haibochen Haibo Chen added a comment - Thanks Peter Backlund for reporting and fixing the issue. The map/reduce counters in JobUnsuccessfulCompletionEvent are inconsistent. In some cases, # of successful mappers is passed to JobUnsuccessfulCompletion.finishedMaps. In other cases, # of successful + failed + killed mappers is reported. I think we should make it as least consistent, and then add failed/killed mapper/reducer count. org.apache.hadoop.mapreduce.v2.app.job.Job is not client facing as far as I can tell, so I think it's fine to add a few more methods (killedMapTaskCount, killedReduceTaskCount). In upgrade cases though, JHS expects the new JobUnsuccessfulCompletion and JobFinishedEvent schema, but it could pick up old .jhist file sthat do not conform with the new schema, we want to make sure it handles the special situation gracefully. Can you check that?
        Hide
        pbacsko Peter Bacsko added a comment -

        Haibo Chen I think you mentioned the wrong user by accident

        Thanks, I'll look at the inconsistent counters.

        I already ran the tests a couple of times and some of them loads existing jhist files, so it should work (values are set to -1 in this case, because it's defined in the Avro schema).

        Show
        pbacsko Peter Bacsko added a comment - Haibo Chen I think you mentioned the wrong user by accident Thanks, I'll look at the inconsistent counters. I already ran the tests a couple of times and some of them loads existing jhist files, so it should work (values are set to -1 in this case, because it's defined in the Avro schema).
        Hide
        haibochen Haibo Chen added a comment -

        Ah ha, sorry about that.

        Show
        haibochen Haibo Chen added a comment - Ah ha, sorry about that.
        Hide
        pbacsko Peter Bacsko added a comment -

        Haibo Chen I addressed your comments regarding the inconsistencies. Also added a new test to verify the number of killed/failed tasks.

        Show
        pbacsko Peter Bacsko added a comment - Haibo Chen I addressed your comments regarding the inconsistencies. Also added a new test to verify the number of killed/failed tasks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        0 patch 0m 2s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        0 mvndep 1m 29s Maven dependency ordering for branch
        +1 mvninstall 13m 8s trunk passed
        +1 compile 7m 57s trunk passed
        +1 checkstyle 1m 38s trunk passed
        +1 mvnsite 1m 33s trunk passed
        +1 mvneclipse 1m 2s trunk passed
        +1 findbugs 2m 23s trunk passed
        +1 javadoc 1m 1s trunk passed
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 1m 23s the patch passed
        +1 compile 7m 56s the patch passed
        +1 javac 7m 56s the patch passed
        +1 checkstyle 1m 39s the patch passed
        +1 mvnsite 1m 34s the patch passed
        +1 mvneclipse 1m 2s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 2m 58s the patch passed
        -1 javadoc 0m 22s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 58 new + 2496 unchanged - 0 fixed = 2554 total (was 2496)
        +1 unit 2m 40s hadoop-mapreduce-client-core in the patch passed.
        +1 unit 8m 50s hadoop-mapreduce-client-app in the patch passed.
        +1 unit 3m 9s hadoop-mapreduce-client-hs in the patch passed.
        +1 unit 0m 20s hadoop-rumen in the patch passed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        64m 38s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872981/MAPREDUCE-6892-002.PATCH
        JIRA Issue MAPREDUCE-6892
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 36f8259e6ab8 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6ed54f3
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6993/artifact/patchprocess/whitespace-eol.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6993/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6993/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6993/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. 0 patch 0m 2s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 1m 29s Maven dependency ordering for branch +1 mvninstall 13m 8s trunk passed +1 compile 7m 57s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 1m 2s trunk passed +1 findbugs 2m 23s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 7m 56s the patch passed +1 javac 7m 56s the patch passed +1 checkstyle 1m 39s the patch passed +1 mvnsite 1m 34s the patch passed +1 mvneclipse 1m 2s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 58s the patch passed -1 javadoc 0m 22s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 58 new + 2496 unchanged - 0 fixed = 2554 total (was 2496) +1 unit 2m 40s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 50s hadoop-mapreduce-client-app in the patch passed. +1 unit 3m 9s hadoop-mapreduce-client-hs in the patch passed. +1 unit 0m 20s hadoop-rumen in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 64m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872981/MAPREDUCE-6892-002.PATCH JIRA Issue MAPREDUCE-6892 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 36f8259e6ab8 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6ed54f3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6993/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6993/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6993/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6993/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        pbacsko Peter Bacsko added a comment -

        Will fix javadoc errors in the next patch.

        Show
        pbacsko Peter Bacsko added a comment - Will fix javadoc errors in the next patch.
        Hide
        haibochen Haibo Chen added a comment -

        Thanks Peter Bacsko for the patch! A few comments:
        1) Now that JobUnsuccessfulCompletionEvent.finishedMaps and JobUnsuccessfulCompletionEvent.finishedReduces are counters for succeeded maps and reduces, can we rename them to succeededMaps and succeededReduces to avoid ambiguity?
        2) While at it, it'd great if we also add the new fields to the corresponding timelineevents in JobUnsuccessfulCompletionEvent.toTimelineEvent().
        3) One more minor issue: remove the redundant new line above JobImpl.unsuccessfulFinish()

        Show
        haibochen Haibo Chen added a comment - Thanks Peter Bacsko for the patch! A few comments: 1) Now that JobUnsuccessfulCompletionEvent.finishedMaps and JobUnsuccessfulCompletionEvent.finishedReduces are counters for succeeded maps and reduces, can we rename them to succeededMaps and succeededReduces to avoid ambiguity? 2) While at it, it'd great if we also add the new fields to the corresponding timelineevents in JobUnsuccessfulCompletionEvent.toTimelineEvent(). 3) One more minor issue: remove the redundant new line above JobImpl.unsuccessfulFinish()
        Hide
        pbacsko Peter Bacsko added a comment -

        Regarding JobUnsuccessfulCompletionEvent - we can certainly rename the fields. But the problem is that the "finishedMaps" and "finishedReduces" are also defined in the Avro schema (Events.avpr). Shall we rename those too? I believe that would introduce a big backward incompatible change.

        I also found inconsistent naming in JobFinishedEvent. Right now it looks like this:

            tEvent.addInfo("NUM_MAPS", getFinishedMaps());
            tEvent.addInfo("NUM_REDUCES", getFinishedReduces());
            tEvent.addInfo("FAILED_MAPS", getFailedMaps());
            tEvent.addInfo("FAILED_REDUCES", getFailedReduces());
            tEvent.addInfo("FINISHED_MAPS", getFinishedMaps());
            tEvent.addInfo("FINISHED_REDUCES", getFinishedReduces());
        

        So NUM_* and FINISHED_* keys contain the same thing.

        Show
        pbacsko Peter Bacsko added a comment - Regarding JobUnsuccessfulCompletionEvent - we can certainly rename the fields. But the problem is that the "finishedMaps" and "finishedReduces" are also defined in the Avro schema (Events.avpr). Shall we rename those too? I believe that would introduce a big backward incompatible change. I also found inconsistent naming in JobFinishedEvent. Right now it looks like this: tEvent.addInfo( "NUM_MAPS" , getFinishedMaps()); tEvent.addInfo( "NUM_REDUCES" , getFinishedReduces()); tEvent.addInfo( "FAILED_MAPS" , getFailedMaps()); tEvent.addInfo( "FAILED_REDUCES" , getFailedReduces()); tEvent.addInfo( "FINISHED_MAPS" , getFinishedMaps()); tEvent.addInfo( "FINISHED_REDUCES" , getFinishedReduces()); So NUM_* and FINISHED_* keys contain the same thing.
        Hide
        haibochen Haibo Chen added a comment -

        Thanks for bringing out the compatibility issue. Let's keep the names and just add new fields.

        Show
        haibochen Haibo Chen added a comment - Thanks for bringing out the compatibility issue. Let's keep the names and just add new fields.
        Hide
        pbacsko Peter Bacsko added a comment - - edited

        Haibo Chen I have been thinking about this renaming and IMO we're better off solve it in a separate JIRA.

        Some problems:
        1. I can add new fields to the Avro schema and keep the old ones, but it adds to the complexity, because we have to support both pairs
        2. We probably also have to check whether "finished" or "succeeded" values are defined in the jhist file, but not both
        3. Looking at the toTimelineEvent() methods, those are a bit messed up too. As I mentioned above, it has prop. keys like NUM_MAPS and FINISHED_MAPS, but both contain the same value. So either we rename NUM_MAPS to SUCCEEDED_MAPS and delete FINISHED_MAPS or NUM_MAPS should be succeeded+failed+killed and rename FINISHED_MAPS to SUCCEEDED_MAPS.

        So it's a bit more complicated. It looks more beneficial to us if we create a separate JIRA where we try to consolidate the naming inconsistencies (and possibly involve other upstream devs) and leave these fields alone in this one. What do you think?

        Show
        pbacsko Peter Bacsko added a comment - - edited Haibo Chen I have been thinking about this renaming and IMO we're better off solve it in a separate JIRA. Some problems: 1. I can add new fields to the Avro schema and keep the old ones, but it adds to the complexity, because we have to support both pairs 2. We probably also have to check whether "finished" or "succeeded" values are defined in the jhist file, but not both 3. Looking at the toTimelineEvent() methods, those are a bit messed up too. As I mentioned above, it has prop. keys like NUM_MAPS and FINISHED_MAPS , but both contain the same value. So either we rename NUM_MAPS to SUCCEEDED_MAPS and delete FINISHED_MAPS or NUM_MAPS should be succeeded+failed+killed and rename FINISHED_MAPS to SUCCEEDED_MAPS . So it's a bit more complicated. It looks more beneficial to us if we create a separate JIRA where we try to consolidate the naming inconsistencies (and possibly involve other upstream devs) and leave these fields alone in this one. What do you think?
        Hide
        haibochen Haibo Chen added a comment -

        3. Looking at the toTimelineEvent() methods, those are a bit messed up too. As I mentioned above, it has prop. keys like NUM_MAPS and FINISHED_MAPS, but both contain the same value. So either we rename NUM_MAPS to SUCCEEDED_MAPS and delete FINISHED_MAPS or NUM_MAPS should be succeeded+failed+killed and rename FINISHED_MAPS to SUCCEEDED_MAPS.

        TimelineServiceV2 related changes (new fields and renames) are fine as long as we do it before 3.0 beta1, from which we'll need to consider compatibility. So let's also include the changes needed here for ATSv2.

        1. I can add new fields to the Avro schema and keep the old ones, but it adds to the complexity, because we have to support both pairs

        2. We probably also have to check whether "finished" or "succeeded" values are defined in the jhist file, but not both

        we can keep the old names and only add new fields, so we continue to use 'finished' as succeeded. In this case, there won't be a new field, succeeded.

        A few more minor comments on the 2rd patch:
        1) In JobHistoryEventHandler when we handles Job_KILL event, can we also add killedMappers + killedReducers in the summary as well?
        2) there is one line above JobImpl.unsuccessfulFinish() which is unnecessary.
        3) In JobImpl.InternalTerminationTransition, can we include all killed/failed counters instead of zeroes?
        4) JobImpl#L1959 is more than 80 characters
        5) In Job20LineHistoryEventEmitter, can we also parse the killed + failed counters instead of passing -1s?
        6) CompletedJob: instead of returning 0s for the new counters, we can return jobInfo.get*() that we get from parsing .jhist files.
        7) In TestJobHistoryParser.testHistoryParsingForKilledAndFailedAttempts(), verification of noOffailedAttempts seems unnecessary.
        In fact, the total # depends on job configuration. Can we remove the related code?

        Show
        haibochen Haibo Chen added a comment - 3. Looking at the toTimelineEvent() methods, those are a bit messed up too. As I mentioned above, it has prop. keys like NUM_MAPS and FINISHED_MAPS, but both contain the same value. So either we rename NUM_MAPS to SUCCEEDED_MAPS and delete FINISHED_MAPS or NUM_MAPS should be succeeded+failed+killed and rename FINISHED_MAPS to SUCCEEDED_MAPS. TimelineServiceV2 related changes (new fields and renames) are fine as long as we do it before 3.0 beta1, from which we'll need to consider compatibility. So let's also include the changes needed here for ATSv2. 1. I can add new fields to the Avro schema and keep the old ones, but it adds to the complexity, because we have to support both pairs 2. We probably also have to check whether "finished" or "succeeded" values are defined in the jhist file, but not both we can keep the old names and only add new fields, so we continue to use 'finished' as succeeded. In this case, there won't be a new field, succeeded. A few more minor comments on the 2rd patch: 1) In JobHistoryEventHandler when we handles Job_KILL event, can we also add killedMappers + killedReducers in the summary as well? 2) there is one line above JobImpl.unsuccessfulFinish() which is unnecessary. 3) In JobImpl.InternalTerminationTransition, can we include all killed/failed counters instead of zeroes? 4) JobImpl#L1959 is more than 80 characters 5) In Job20LineHistoryEventEmitter, can we also parse the killed + failed counters instead of passing -1s? 6) CompletedJob: instead of returning 0s for the new counters, we can return jobInfo.get*() that we get from parsing .jhist files. 7) In TestJobHistoryParser.testHistoryParsingForKilledAndFailedAttempts(), verification of noOffailedAttempts seems unnecessary. In fact, the total # depends on job configuration. Can we remove the related code?
        Hide
        pbacsko Peter Bacsko added a comment -

        we can keep the old names and only add new fields, so we continue to use 'finished' as succeeded. In this case, there won't be a new field, succeeded.

        You mean we keep the schema but rename stuff in the class from "finished" to "succeeded", including methods? That's certainly a lot easier.
        The names "finishedMaps" and "finishedReduces" occur at other parts of the code. If you search for either "finishedMaps" or "finishedReduces" in GitHub, you get 16 results:
        https://github.com/apache/hadoop/search?p=1&q=finishedmaps&type=&utf8=%E2%9C%93

        Shall we go ahead and do the rename where it's necessary? (there could be some exceptions like ShuffleSchedulerImpl.java)

        5) In Job20LineHistoryEventEmitter, can we also parse the killed + failed counters instead of passing -1s?

        That's some old code in Apache Rumen. I'm not exactly sure what is parses but I doubt that would contain string like KILLED_MAPS or KILLED_REDUCES.

        7) In TestJobHistoryParser.testHistoryParsingForKilledAndFailedAttempts(), verification of noOffailedAttempts seems unnecessary.

        That was a copy-paste from an existing testcase testHistoryParsingForFailedAttempts(). You sure we don't need that?

        Show
        pbacsko Peter Bacsko added a comment - we can keep the old names and only add new fields, so we continue to use 'finished' as succeeded. In this case, there won't be a new field, succeeded. You mean we keep the schema but rename stuff in the class from "finished" to "succeeded", including methods? That's certainly a lot easier. The names "finishedMaps" and "finishedReduces" occur at other parts of the code. If you search for either "finishedMaps" or "finishedReduces" in GitHub, you get 16 results: https://github.com/apache/hadoop/search?p=1&q=finishedmaps&type=&utf8=%E2%9C%93 Shall we go ahead and do the rename where it's necessary? (there could be some exceptions like ShuffleSchedulerImpl.java) 5) In Job20LineHistoryEventEmitter, can we also parse the killed + failed counters instead of passing -1s? That's some old code in Apache Rumen. I'm not exactly sure what is parses but I doubt that would contain string like KILLED_MAPS or KILLED_REDUCES. 7) In TestJobHistoryParser.testHistoryParsingForKilledAndFailedAttempts(), verification of noOffailedAttempts seems unnecessary. That was a copy-paste from an existing testcase testHistoryParsingForFailedAttempts() . You sure we don't need that?
        Hide
        haibochen Haibo Chen added a comment - - edited

        Shall we go ahead and do the rename where it's necessary?

        I think yes for all source code references whenever the field name does not correspond to what it really represents , but not the schema for compatibility purpose. For schema, if possible, we can add a comment there for clarity

        I'm not exactly sure what is parses but I doubt that would contain string like KILLED_MAPS or KILLED_REDUCES.

        -1 in this context means unavailable or unknown. Job20LineHistoryEventEmitter is where we generate some of the events. I guess it really does not matter, so let's keep -1s.

        That was a copy-paste from an existing testcase testHistoryParsingForFailedAttempts(). You sure we don't need that?

        MRAppWithHistoryWithFailedAttempt always kills the 1st map task and fails the 1st reduce task. We want to test if the failed/killed/succeeded task counters are correct/ # of failed/killed task attempts are not directly related and in addition, how many attempts each task can have is configurable, may not be 2 necessarily. Thus, I suggested getting rid of noOffailedAttempts, adding assertion of # of succeeded tasks.

        Show
        haibochen Haibo Chen added a comment - - edited Shall we go ahead and do the rename where it's necessary? I think yes for all source code references whenever the field name does not correspond to what it really represents , but not the schema for compatibility purpose. For schema, if possible, we can add a comment there for clarity I'm not exactly sure what is parses but I doubt that would contain string like KILLED_MAPS or KILLED_REDUCES. -1 in this context means unavailable or unknown. Job20LineHistoryEventEmitter is where we generate some of the events. I guess it really does not matter, so let's keep -1s. That was a copy-paste from an existing testcase testHistoryParsingForFailedAttempts(). You sure we don't need that? MRAppWithHistoryWithFailedAttempt always kills the 1st map task and fails the 1st reduce task. We want to test if the failed/killed/succeeded task counters are correct/ # of failed/killed task attempts are not directly related and in addition, how many attempts each task can have is configurable, may not be 2 necessarily. Thus, I suggested getting rid of noOffailedAttempts, adding assertion of # of succeeded tasks.
        Hide
        pbacsko Peter Bacsko added a comment -

        Haibo Chen I uploaded patch v3 which addresses your comments.

        I think there are still some room for improvements, plus I expect checkstyle + javadoc warnings which needs to be fixed too.

        Show
        pbacsko Peter Bacsko added a comment - Haibo Chen I uploaded patch v3 which addresses your comments. I think there are still some room for improvements, plus I expect checkstyle + javadoc warnings which needs to be fixed too.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 33s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
              trunk Compile Tests
        0 mvndep 1m 30s Maven dependency ordering for branch
        +1 mvninstall 16m 15s trunk passed
        +1 compile 15m 48s trunk passed
        +1 checkstyle 2m 19s trunk passed
        +1 mvnsite 2m 6s trunk passed
        +1 findbugs 3m 1s trunk passed
        +1 javadoc 1m 35s trunk passed
              Patch Compile Tests
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 43s the patch passed
        +1 compile 15m 55s the patch passed
        +1 javac 15m 55s the patch passed
        -1 checkstyle 2m 43s root: The patch generated 67 new + 1076 unchanged - 27 fixed = 1143 total (was 1103)
        +1 mvnsite 2m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 16s the patch passed
        -1 javadoc 0m 39s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 58 new + 2496 unchanged - 0 fixed = 2554 total (was 2496)
              Other Tests
        +1 unit 3m 19s hadoop-mapreduce-client-core in the patch passed.
        +1 unit 9m 56s hadoop-mapreduce-client-app in the patch passed.
        -1 unit 4m 3s hadoop-mapreduce-client-hs in the patch failed.
        +1 unit 0m 33s hadoop-rumen in the patch passed.
        +1 asflicense 0m 39s The patch does not generate ASF License warnings.
        93m 1s



        Reason Tests
        Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryParsing



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue MAPREDUCE-6892
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881446/MAPREDUCE-6892-003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b2c19b2fa258 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d964062
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/artifact/patchprocess/diff-checkstyle-root.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/console
        Powered by Apache Yetus 0.5.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 1m 30s Maven dependency ordering for branch +1 mvninstall 16m 15s trunk passed +1 compile 15m 48s trunk passed +1 checkstyle 2m 19s trunk passed +1 mvnsite 2m 6s trunk passed +1 findbugs 3m 1s trunk passed +1 javadoc 1m 35s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 43s the patch passed +1 compile 15m 55s the patch passed +1 javac 15m 55s the patch passed -1 checkstyle 2m 43s root: The patch generated 67 new + 1076 unchanged - 27 fixed = 1143 total (was 1103) +1 mvnsite 2m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 16s the patch passed -1 javadoc 0m 39s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 58 new + 2496 unchanged - 0 fixed = 2554 total (was 2496)       Other Tests +1 unit 3m 19s hadoop-mapreduce-client-core in the patch passed. +1 unit 9m 56s hadoop-mapreduce-client-app in the patch passed. -1 unit 4m 3s hadoop-mapreduce-client-hs in the patch failed. +1 unit 0m 33s hadoop-rumen in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 93m 1s Reason Tests Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryParsing Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue MAPREDUCE-6892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881446/MAPREDUCE-6892-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b2c19b2fa258 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d964062 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7060/console Powered by Apache Yetus 0.5.0 http://yetus.apache.org This message was automatically generated.
        Hide
        pbacsko Peter Bacsko added a comment -

        Test case passes locally, I believe there is a race condition somewhere.

        Show
        pbacsko Peter Bacsko added a comment - Test case passes locally, I believe there is a race condition somewhere.
        Hide
        haibochen Haibo Chen added a comment -

        Thanks Peter Bacsko for updating the patch! A few more comments:
        1) In JobFinishedEvent, can we add code in getDatum and setDatum() to handle the newly added fields?
        2) In UnparsedJob, let return -1 to be consistent with getCompletedMaps() and getCompletedReduce(). Similarly for PartialJob, let's also return -1 to indicate the info is not available.
        3) JobImpl.getCompletedMaps() return successMap + killedMap + FailedMap, whereas CompletedJob.getCompletedMaps() returns only successMap.
        Let's do the same in CompletedJob.getCompletedMaps() as well as in CompletedJob.getCompletedReduce().
        4) in Job20LineHistoryEventEmitter, how much work is it to also parse the failed/killed map/reducer counters (I have not familiar with this code)? I am OK to leave it if it is too much.
        5) Not an issue with this patch, but let's also set killed/failed counters in JobHistoryParser.handleJobFinishedEvent()
        6) CompletedJob.getKillReduces() should return (int) jobInfo.getKilledReduces();
        7) Rename JobSummary.getNumFinishedMaps() to getNumSucceededMaps(). Also, let's add summary.setNumKilled[Map/Reduce] in TestJobSummary.before() as well.

        Can you look into the test failure and fix it if possible?

        Show
        haibochen Haibo Chen added a comment - Thanks Peter Bacsko for updating the patch! A few more comments: 1) In JobFinishedEvent, can we add code in getDatum and setDatum() to handle the newly added fields? 2) In UnparsedJob, let return -1 to be consistent with getCompletedMaps() and getCompletedReduce(). Similarly for PartialJob, let's also return -1 to indicate the info is not available. 3) JobImpl.getCompletedMaps() return successMap + killedMap + FailedMap, whereas CompletedJob.getCompletedMaps() returns only successMap. Let's do the same in CompletedJob.getCompletedMaps() as well as in CompletedJob.getCompletedReduce(). 4) in Job20LineHistoryEventEmitter, how much work is it to also parse the failed/killed map/reducer counters (I have not familiar with this code)? I am OK to leave it if it is too much. 5) Not an issue with this patch, but let's also set killed/failed counters in JobHistoryParser.handleJobFinishedEvent() 6) CompletedJob.getKillReduces() should return (int) jobInfo.getKilledReduces(); 7) Rename JobSummary.getNumFinishedMaps() to getNumSucceededMaps(). Also, let's add summary.setNumKilled [Map/Reduce] in TestJobSummary.before() as well. Can you look into the test failure and fix it if possible?
        Hide
        pbacsko Peter Bacsko added a comment -

        Thanks a lot Haibo Chen for the comments, this renaming thing is tricky

        In the next iteration I'll address the issues you mentioned above, plus the javadoc + checkstyle problems and I'll try to fix the test.

        Show
        pbacsko Peter Bacsko added a comment - Thanks a lot Haibo Chen for the comments, this renaming thing is tricky In the next iteration I'll address the issues you mentioned above, plus the javadoc + checkstyle problems and I'll try to fix the test.
        Hide
        pbacsko Peter Bacsko added a comment - - edited

        Haibo Chen I uploaded patch v4. I fixed everything you mentioned, but there are some things which I didn't change:

        1) Checkstyle, Javadoc errors: we would end up with a messy, inconsistent code style if we decide to eliminate these. I kept the style of the existing code, which unfortunately introduced new warnings. Refactoring/re-indenting the existing code would be way too much change and it's unnecessary IMHO.

        2) Job20LineHistoryEventEmitter - I kept the "-1" values. I was looking for the code which generates the stuff that this class parses. I couldn't find it and I believe it does not exist anymore. However I found this on an old branch: https://github.com/apache/hadoop/blob/branch-1.2/src/mapred/org/apache/hadoop/mapred/JobHistory.java

        I think this was the class which generated the events that Apache Rumen is meant to parse. This makes me wonder whether it makes sense to have Rumen on the trunk. Looking at the latest changes, the vast majority of the source files haven't been touched for 5-6 years (and newer changes are completely unrelated to its functionality). At least we should consider deprecating it.

        Show
        pbacsko Peter Bacsko added a comment - - edited Haibo Chen I uploaded patch v4. I fixed everything you mentioned, but there are some things which I didn't change: 1) Checkstyle, Javadoc errors: we would end up with a messy, inconsistent code style if we decide to eliminate these. I kept the style of the existing code, which unfortunately introduced new warnings. Refactoring/re-indenting the existing code would be way too much change and it's unnecessary IMHO. 2) Job20LineHistoryEventEmitter - I kept the "-1" values. I was looking for the code which generates the stuff that this class parses. I couldn't find it and I believe it does not exist anymore. However I found this on an old branch: https://github.com/apache/hadoop/blob/branch-1.2/src/mapred/org/apache/hadoop/mapred/JobHistory.java I think this was the class which generated the events that Apache Rumen is meant to parse. This makes me wonder whether it makes sense to have Rumen on the trunk. Looking at the latest changes, the vast majority of the source files haven't been touched for 5-6 years (and newer changes are completely unrelated to its functionality). At least we should consider deprecating it.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 28s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
              trunk Compile Tests
        0 mvndep 1m 38s Maven dependency ordering for branch
        +1 mvninstall 25m 44s trunk passed
        +1 compile 25m 47s trunk passed
        +1 checkstyle 3m 17s trunk passed
        +1 mvnsite 4m 1s trunk passed
        +1 findbugs 5m 11s trunk passed
        +1 javadoc 2m 52s trunk passed
              Patch Compile Tests
        0 mvndep 0m 23s Maven dependency ordering for patch
        +1 mvninstall 2m 39s the patch passed
        +1 compile 19m 33s the patch passed
        +1 javac 19m 33s the patch passed
        -1 checkstyle 3m 22s root: The patch generated 57 new + 1071 unchanged - 32 fixed = 1128 total (was 1103)
        +1 mvnsite 3m 47s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 6m 19s the patch passed
        -1 javadoc 0m 51s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 50 new + 2496 unchanged - 0 fixed = 2546 total (was 2496)
              Other Tests
        +1 unit 4m 10s hadoop-mapreduce-client-core in the patch passed.
        +1 unit 12m 15s hadoop-mapreduce-client-app in the patch passed.
        -1 unit 5m 35s hadoop-mapreduce-client-hs in the patch failed.
        +1 unit 0m 50s hadoop-rumen in the patch passed.
        +1 asflicense 0m 47s The patch does not generate ASF License warnings.
        134m 19s



        Reason Tests
        Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryEntities



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue MAPREDUCE-6892
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882872/MAPREDUCE-6892-004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d5fd34161e16 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 267e19a
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/artifact/patchprocess/diff-checkstyle-root.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/console
        Powered by Apache Yetus 0.5.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 1m 38s Maven dependency ordering for branch +1 mvninstall 25m 44s trunk passed +1 compile 25m 47s trunk passed +1 checkstyle 3m 17s trunk passed +1 mvnsite 4m 1s trunk passed +1 findbugs 5m 11s trunk passed +1 javadoc 2m 52s trunk passed       Patch Compile Tests 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 2m 39s the patch passed +1 compile 19m 33s the patch passed +1 javac 19m 33s the patch passed -1 checkstyle 3m 22s root: The patch generated 57 new + 1071 unchanged - 32 fixed = 1128 total (was 1103) +1 mvnsite 3m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 19s the patch passed -1 javadoc 0m 51s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 50 new + 2496 unchanged - 0 fixed = 2546 total (was 2496)       Other Tests +1 unit 4m 10s hadoop-mapreduce-client-core in the patch passed. +1 unit 12m 15s hadoop-mapreduce-client-app in the patch passed. -1 unit 5m 35s hadoop-mapreduce-client-hs in the patch failed. +1 unit 0m 50s hadoop-rumen in the patch passed. +1 asflicense 0m 47s The patch does not generate ASF License warnings. 134m 19s Reason Tests Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryEntities Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue MAPREDUCE-6892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882872/MAPREDUCE-6892-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d5fd34161e16 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 267e19a Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7089/console Powered by Apache Yetus 0.5.0 http://yetus.apache.org This message was automatically generated.
        Hide
        pbacsko Peter Bacsko added a comment - - edited

        Looks like I introduced test failures - will look into them tomorrow.

        Update: default value for killed/failed should be 0 and not "-1" in the schema.

        Show
        pbacsko Peter Bacsko added a comment - - edited Looks like I introduced test failures - will look into them tomorrow. Update: default value for killed/failed should be 0 and not "-1" in the schema.
        Hide
        haibochen Haibo Chen added a comment -

        Thanks @Peter for the update, I'll wait for you to fix the test failure before I review.
        I do think the default value for killed failed should be -1 in the schema. -1 means not available/known IIUC.
        In our case, the killed and failed counters were not recorded previously, thus I think -1 makes more sense.

        Show
        haibochen Haibo Chen added a comment - Thanks @Peter for the update, I'll wait for you to fix the test failure before I review. I do think the default value for killed failed should be -1 in the schema. -1 means not available/known IIUC. In our case, the killed and failed counters were not recorded previously, thus I think -1 makes more sense.
        Hide
        pbacsko Peter Bacsko added a comment -

        Uploaded patch v5 where I fixed the flaky test + the other failures.

        Show
        pbacsko Peter Bacsko added a comment - Uploaded patch v5 where I fixed the flaky test + the other failures.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 27s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
              trunk Compile Tests
        0 mvndep 1m 32s Maven dependency ordering for branch
        +1 mvninstall 16m 23s trunk passed
        +1 compile 17m 47s trunk passed
        +1 checkstyle 2m 10s trunk passed
        +1 mvnsite 2m 18s trunk passed
        +1 findbugs 3m 19s trunk passed
        +1 javadoc 1m 42s trunk passed
              Patch Compile Tests
        0 mvndep 0m 18s Maven dependency ordering for patch
        +1 mvninstall 1m 49s the patch passed
        +1 compile 14m 34s the patch passed
        +1 javac 14m 34s the patch passed
        -1 checkstyle 2m 15s root: The patch generated 57 new + 1072 unchanged - 32 fixed = 1129 total (was 1104)
        +1 mvnsite 2m 34s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 6m 6s the patch passed
        -1 javadoc 0m 36s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 50 new + 2496 unchanged - 0 fixed = 2546 total (was 2496)
              Other Tests
        -1 unit 5m 21s hadoop-mapreduce-client-core in the patch failed.
        +1 unit 15m 22s hadoop-mapreduce-client-app in the patch passed.
        +1 unit 6m 52s hadoop-mapreduce-client-hs in the patch passed.
        +1 unit 0m 38s hadoop-rumen in the patch passed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        106m 15s



        Reason Tests
        Failed junit tests hadoop.mapred.TestJobEndNotifier



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue MAPREDUCE-6892
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884030/MAPREDUCE-6892-005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0677a3bc5bd0 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 02599bd
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/artifact/patchprocess/diff-checkstyle-root.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/console
        Powered by Apache Yetus 0.5.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 16m 23s trunk passed +1 compile 17m 47s trunk passed +1 checkstyle 2m 10s trunk passed +1 mvnsite 2m 18s trunk passed +1 findbugs 3m 19s trunk passed +1 javadoc 1m 42s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 49s the patch passed +1 compile 14m 34s the patch passed +1 javac 14m 34s the patch passed -1 checkstyle 2m 15s root: The patch generated 57 new + 1072 unchanged - 32 fixed = 1129 total (was 1104) +1 mvnsite 2m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 6s the patch passed -1 javadoc 0m 36s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 50 new + 2496 unchanged - 0 fixed = 2546 total (was 2496)       Other Tests -1 unit 5m 21s hadoop-mapreduce-client-core in the patch failed. +1 unit 15m 22s hadoop-mapreduce-client-app in the patch passed. +1 unit 6m 52s hadoop-mapreduce-client-hs in the patch passed. +1 unit 0m 38s hadoop-rumen in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 106m 15s Reason Tests Failed junit tests hadoop.mapred.TestJobEndNotifier Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue MAPREDUCE-6892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884030/MAPREDUCE-6892-005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0677a3bc5bd0 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 02599bd Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7099/console Powered by Apache Yetus 0.5.0 http://yetus.apache.org This message was automatically generated.
        Hide
        haibochen Haibo Chen added a comment -

        Thanks for the update Peter Bacsko! The patch is very close to be ready. But I still think we should give -1 as the default value to stand for unknown in the schema (Events.avpr).
        In TestJobHistoryParsing.testHistoryParsingForKilledAndFailedAttempts(), let's use // for the comments you've added (/**/ is for class/methods) and remove the nested for loop to assert RACK NAME for each of the tasks (I don't think they are relevant). Otherwise, the patch LGTM.

        Show
        haibochen Haibo Chen added a comment - Thanks for the update Peter Bacsko ! The patch is very close to be ready. But I still think we should give -1 as the default value to stand for unknown in the schema (Events.avpr). In TestJobHistoryParsing.testHistoryParsingForKilledAndFailedAttempts(), let's use // for the comments you've added (/**/ is for class/methods) and remove the nested for loop to assert RACK NAME for each of the tasks (I don't think they are relevant). Otherwise, the patch LGTM.
        Hide
        pbacsko Peter Bacsko added a comment -

        I still think we should give -1 as the default value to stand for unknown in the schema

        I changed this. However, in order to have the tests pass, I added some extra behavior to CompletedJob, please check that.

        Testcase was modified as well.

        Show
        pbacsko Peter Bacsko added a comment - I still think we should give -1 as the default value to stand for unknown in the schema I changed this. However, in order to have the tests pass, I added some extra behavior to CompletedJob, please check that. Testcase was modified as well.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
              trunk Compile Tests
        0 mvndep 1m 37s Maven dependency ordering for branch
        +1 mvninstall 17m 18s trunk passed
        +1 compile 18m 41s trunk passed
        +1 checkstyle 2m 18s trunk passed
        +1 mvnsite 2m 13s trunk passed
        +1 findbugs 3m 24s trunk passed
        +1 javadoc 1m 29s trunk passed
              Patch Compile Tests
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 42s the patch passed
        +1 compile 14m 18s the patch passed
        +1 javac 14m 18s the patch passed
        -1 checkstyle 2m 16s root: The patch generated 57 new + 1072 unchanged - 32 fixed = 1129 total (was 1104)
        +1 mvnsite 2m 7s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 5s the patch passed
        -1 javadoc 0m 30s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 50 new + 2496 unchanged - 0 fixed = 2546 total (was 2496)
              Other Tests
        +1 unit 3m 19s hadoop-mapreduce-client-core in the patch passed.
        +1 unit 10m 22s hadoop-mapreduce-client-app in the patch passed.
        +1 unit 3m 56s hadoop-mapreduce-client-hs in the patch passed.
        +1 unit 0m 29s hadoop-rumen in the patch passed.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        93m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue MAPREDUCE-6892
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884267/MAPREDUCE-6892-006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e6186c2b2dd3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 33afa94
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7105/artifact/patchprocess/diff-checkstyle-root.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7105/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7105/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7105/console
        Powered by Apache Yetus 0.5.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 1m 37s Maven dependency ordering for branch +1 mvninstall 17m 18s trunk passed +1 compile 18m 41s trunk passed +1 checkstyle 2m 18s trunk passed +1 mvnsite 2m 13s trunk passed +1 findbugs 3m 24s trunk passed +1 javadoc 1m 29s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 42s the patch passed +1 compile 14m 18s the patch passed +1 javac 14m 18s the patch passed -1 checkstyle 2m 16s root: The patch generated 57 new + 1072 unchanged - 32 fixed = 1129 total (was 1104) +1 mvnsite 2m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 5s the patch passed -1 javadoc 0m 30s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 50 new + 2496 unchanged - 0 fixed = 2546 total (was 2496)       Other Tests +1 unit 3m 19s hadoop-mapreduce-client-core in the patch passed. +1 unit 10m 22s hadoop-mapreduce-client-app in the patch passed. +1 unit 3m 56s hadoop-mapreduce-client-hs in the patch passed. +1 unit 0m 29s hadoop-rumen in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 93m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue MAPREDUCE-6892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884267/MAPREDUCE-6892-006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e6186c2b2dd3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 33afa94 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7105/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7105/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7105/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-rumen U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7105/console Powered by Apache Yetus 0.5.0 http://yetus.apache.org This message was automatically generated.
        Hide
        haibochen Haibo Chen added a comment -

        +1 on the latest patch. Will commit it shortly.

        Show
        haibochen Haibo Chen added a comment - +1 on the latest patch. Will commit it shortly.
        Hide
        haibochen Haibo Chen added a comment -

        Thanks Peter Bacsko for the contribution, I have committed the patch to trunk

        Show
        haibochen Haibo Chen added a comment - Thanks Peter Bacsko for the contribution, I have committed the patch to trunk
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12275 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12275/)
        MAPREDUCE-6892. Issues with the count of failed/killed tasks in the (haibochen: rev d04f85f387e4a78816bc9966ee2b4a647ee05faf)

        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
        • (edit) hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Job20LineHistoryEventEmitter.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAcls.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobUnsuccessfulCompletionEvent.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JSONHistoryViewerPrinter.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobFinishedEvent.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestHistoryViewerPrinter.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HumanReadableHistoryViewerPrinter.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/UnparsedJob.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12275 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12275/ ) MAPREDUCE-6892 . Issues with the count of failed/killed tasks in the (haibochen: rev d04f85f387e4a78816bc9966ee2b4a647ee05faf) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java (edit) hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Job20LineHistoryEventEmitter.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAcls.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobUnsuccessfulCompletionEvent.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JSONHistoryViewerPrinter.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobFinishedEvent.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestHistoryViewerPrinter.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HumanReadableHistoryViewerPrinter.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/UnparsedJob.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Peter, do you mind adding a release note to this JIRA summarizing the impact for our end users? Thanks!

        Show
        andrew.wang Andrew Wang added a comment - Peter, do you mind adding a release note to this JIRA summarizing the impact for our end users? Thanks!

          People

          • Assignee:
            pbacsko Peter Bacsko
            Reporter:
            pbacsko Peter Bacsko
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development