Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4025

AM can crash if task attempt reports bogus progress value

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.2
    • Fix Version/s: 0.23.2
    • Component/s: mr-am, mrv2
    • Labels:
      None

      Description

      If a task attempt reports a bogus progress value (e.g.: something above 1.0) then the AM can crash like this:

      java.lang.ArrayIndexOutOfBoundsException: 12
      	at org.apache.hadoop.mapred.PeriodicStatsAccumulator.extend(PeriodicStatsAccumulator.java:185)
      	at org.apache.hadoop.mapred.WrappedPeriodicStatsAccumulator.extend(WrappedPeriodicStatsAccumulator.java:31)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.updateProgressSplits(TaskAttemptImpl.java:1043)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$4100(TaskAttemptImpl.java:136)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1509)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1490)
      	at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
      	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
      	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
      	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:931)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:886)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:878)
      	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
      	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
      	at java.lang.Thread.run(Thread.java:619)
      

        Activity

        Hide
        Jason Lowe added a comment -

        Patch to simply clamp the progress value to 0.0-1.0 before we try to update the counters.

        Needs a test. I've manually verified the fix.

        Show
        Jason Lowe added a comment - Patch to simply clamp the progress value to 0.0-1.0 before we try to update the counters. Needs a test. I've manually verified the fix.
        Hide
        Robert Joseph Evans added a comment -

        The patch looks good to me +1, assuming that Jenkins comes back with no issues. I normally would like to see a unit test for this, but it is just a one line change that is very simple to understand so I am OK with it.

        Show
        Robert Joseph Evans added a comment - The patch looks good to me +1, assuming that Jenkins comes back with no issues. I normally would like to see a unit test for this, but it is just a one line change that is very simple to understand so I am OK with it.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12518749/MAPREDUCE-4025.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2070//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2070//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518749/MAPREDUCE-4025.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2070//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2070//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Jason. I just checked this into trunk, 0.23 and 0.23.2

        Show
        Robert Joseph Evans added a comment - Thanks Jason. I just checked this into trunk, 0.23 and 0.23.2
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1901 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1901/)
        MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302645)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302645
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1901 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1901/ ) MAPREDUCE-4025 . AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302645) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302645 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1975 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1975/)
        MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302645)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302645
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1975 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1975/ ) MAPREDUCE-4025 . AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302645) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302645 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #703 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/703/)
        svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302646)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302646
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #703 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/703/ ) svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025 . AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302646) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302646 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #694 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/694/)
        svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302646)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302646
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #694 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/694/ ) svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025 . AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302646) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302646 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1909 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1909/)
        MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302645)

        Result = ABORTED
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302645
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1909 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1909/ ) MAPREDUCE-4025 . AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302645) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302645 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #710 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/710/)
        svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302646)

        Result = ABORTED
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302646
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #710 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/710/ ) svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025 . AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302646) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302646 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #231 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/)
        svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302646)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302646
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #231 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/ ) svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025 . AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302646) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302646 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/)
        MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302645)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302645
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/ ) MAPREDUCE-4025 . AM can crash if task attempt reports bogus progress value (Jason Lowe via bobby) (Revision 1302645) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302645 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development