Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5817

Mappers get rescheduled on node transition even after all reducers are completed

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.8.0
    • Component/s: applicationmaster
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We're seeing a behavior where a job runs long after all reducers were already finished. We found that the job was rescheduling and running a number of mappers beyond the point of reducer completion. In one situation, the job ran for some 9 more hours after all reducers completed!

      This happens because whenever a node transition (to an unusable state) comes into the app master, it just reschedules all mappers that already ran on the node in all cases.

      Therefore, if any node transition has a potential to extend the job period. Once this window opens, another node transition can prolong it, and this can happen indefinitely in theory.

      If there is some instability in the pool (unhealthy, etc.) for a duration, then any big job is severely vulnerable to this problem.

      If all reducers have been completed, JobImpl.actOnUnusableNode() should not reschedule mapper tasks. If all reducers are completed, the mapper outputs are no longer needed, and there is no need to reschedule mapper tasks as they would not be consumed anyway.

      1. mapreduce-5817.patch
        13 kB
        Sangjin Lee
      2. MAPREDUCE-5817.002.patch
        12 kB
        Sangjin Lee
      3. MAPREDUCE-5817.001.patch
        13 kB
        Sangjin Lee

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/277/)
          MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/277/ ) MAPREDUCE-5817 . Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2215 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2215/)
          MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2215 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2215/ ) MAPREDUCE-5817 . Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #285 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/285/)
          MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #285 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/285/ ) MAPREDUCE-5817 . Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2234 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2234/)
          MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2234 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2234/ ) MAPREDUCE-5817 . Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1018 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1018/)
          MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1018 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1018/ ) MAPREDUCE-5817 . Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/288/)
          MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/288/ ) MAPREDUCE-5817 . Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          Sangjin Lee added a comment -

          Thanks Chris Douglas for the review and Karthik Kambatla for the commit!

          Show
          Sangjin Lee added a comment - Thanks Chris Douglas for the review and Karthik Kambatla for the commit!
          Hide
          Karthik Kambatla added a comment -

          Just committed this to trunk and branch-2.

          Thanks Sangjin Lee for the contribution and Chris Douglas for the review.

          Show
          Karthik Kambatla added a comment - Just committed this to trunk and branch-2. Thanks Sangjin Lee for the contribution and Chris Douglas for the review.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8306 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8306/)
          MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8306 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8306/ ) MAPREDUCE-5817 . Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) (kasha: rev 27d24f96ab8d17e839a1ef0d7076efc78d28724a) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          Hide
          Karthik Kambatla added a comment -

          TestJobEndNotifier passes locally.

          +1, checking this in.

          Show
          Karthik Kambatla added a comment - TestJobEndNotifier passes locally. +1, checking this in.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 2s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 51s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 33s The applied patch generated 1 new checkstyle issues (total was 108, now 107).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 7s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 8m 56s Tests failed in hadoop-mapreduce-client-app.
              46m 33s  



          Reason Tests
          Failed unit tests hadoop.mapreduce.v2.app.TestJobEndNotifier



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750568/MAPREDUCE-5817.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 84bf712
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5936/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5936/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5936/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5936/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 2s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 51s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 33s The applied patch generated 1 new checkstyle issues (total was 108, now 107). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 7s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 8m 56s Tests failed in hadoop-mapreduce-client-app.     46m 33s   Reason Tests Failed unit tests hadoop.mapreduce.v2.app.TestJobEndNotifier Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750568/MAPREDUCE-5817.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 84bf712 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5936/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5936/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5936/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5936/console This message was automatically generated.
          Hide
          Sangjin Lee added a comment -

          v.2 patch posted.

          I changed allReducersComplete() to use getCompletedReduces(). Thanks for the suggestion. Strictly speaking, the first check numReduceTasks == 0 is bit redundant, but it's just to be explicit for map-only jobs.

          Show
          Sangjin Lee added a comment - v.2 patch posted. I changed allReducersComplete() to use getCompletedReduces() . Thanks for the suggestion. Strictly speaking, the first check numReduceTasks == 0 is bit redundant, but it's just to be explicit for map-only jobs.
          Hide
          Ben Podgursky added a comment -

          Thanks Sangjin, that looks good to me (not that I'm super familiar with the hadoop source).

          Show
          Ben Podgursky added a comment - Thanks Sangjin, that looks good to me (not that I'm super familiar with the hadoop source).
          Hide
          Chris Douglas added a comment -

          The current patch skips re-running mappers only if all reducers are complete. So I don't think reducers will fail beyond that point? Did I understand your question right?

          I see; sorry, I hadn't read the rest of the JIRA carefully. That's a fairly narrow window, isn't it? We may not need an extra state, if we kill all running maps when the last reducer completes. The condition this adds prevents new maps from being scheduled while cleanup/commit code is running.

          Minor: could allReducersComplete() call getCompletedReduces()?

          +1 on the patch

          Show
          Chris Douglas added a comment - The current patch skips re-running mappers only if all reducers are complete. So I don't think reducers will fail beyond that point? Did I understand your question right? I see; sorry, I hadn't read the rest of the JIRA carefully. That's a fairly narrow window, isn't it? We may not need an extra state, if we kill all running maps when the last reducer completes. The condition this adds prevents new maps from being scheduled while cleanup/commit code is running. Minor: could allReducersComplete() call getCompletedReduces() ? +1 on the patch
          Hide
          Sangjin Lee added a comment -

          The test failures appear unrelated. The checkstyle is about the length of file JobImpl.java which is pretty much an existing issue.

          Show
          Sangjin Lee added a comment - The test failures appear unrelated. The checkstyle is about the length of file JobImpl.java which is pretty much an existing issue.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 50s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 11s There were no new javac warning messages.
          +1 javadoc 10m 4s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 35s The applied patch generated 1 new checkstyle issues (total was 108, now 107).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 1m 10s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 9m 6s Tests failed in hadoop-mapreduce-client-app.
              48m 26s  



          Reason Tests
          Failed unit tests hadoop.mapreduce.v2.app.TestJobEndNotifier



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749938/MAPREDUCE-5817.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7c796fd
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5934/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5934/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5934/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5934/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 50s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 11s There were no new javac warning messages. +1 javadoc 10m 4s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 35s The applied patch generated 1 new checkstyle issues (total was 108, now 107). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 1m 10s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 9m 6s Tests failed in hadoop-mapreduce-client-app.     48m 26s   Reason Tests Failed unit tests hadoop.mapreduce.v2.app.TestJobEndNotifier Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749938/MAPREDUCE-5817.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7c796fd checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5934/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5934/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5934/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5934/console This message was automatically generated.
          Hide
          Sangjin Lee added a comment -

          The current patch skips re-running mappers only if all reducers are complete. So I don't think reducers will fail beyond that point? Did I understand your question right?

          Show
          Sangjin Lee added a comment - The current patch skips re-running mappers only if all reducers are complete. So I don't think reducers will fail beyond that point? Did I understand your question right?
          Hide
          Chris Douglas added a comment -

          Does this work if the reducer fails subsequently? Presumably reexecution will be triggered by fetch failures?

          Show
          Chris Douglas added a comment - Does this work if the reducer fails subsequently? Presumably reexecution will be triggered by fetch failures?
          Hide
          Sangjin Lee added a comment -

          v.1 patch posted.

          This implements option (1).

          Show
          Sangjin Lee added a comment - v.1 patch posted. This implements option (1).
          Hide
          Ben Podgursky added a comment -

          Option (2) sounds a lot trickier (and more of a risk tradeoff) because tasks do realistically die or get pre-empted after copying, and it could hurt runtime to * not * restart tasks in case that happens. So you'd want that to be configurable either way. In an ideal world, I would just kill all map tasks which were running at the time the last reduce finished... but I don't know the implications this would have for counters, etc.

          So I'd vote (1) for now since I think it's pure upside, and leave (2) for a later JIRA since involves more tradeoffs.

          Show
          Ben Podgursky added a comment - Option (2) sounds a lot trickier (and more of a risk tradeoff) because tasks do realistically die or get pre-empted after copying, and it could hurt runtime to * not * restart tasks in case that happens. So you'd want that to be configurable either way. In an ideal world, I would just kill all map tasks which were running at the time the last reduce finished... but I don't know the implications this would have for counters, etc. So I'd vote (1) for now since I think it's pure upside, and leave (2) for a later JIRA since involves more tradeoffs.
          Hide
          Sangjin Lee added a comment -

          Thanks for the comments Ben Podgursky. I've been meaning to revisit this JIRA.

          This JIRA is stalled because there are 2 options to move forward with different implications. Option (1) is to prevent re-running mappers as soon as all reducers complete, and the option (2) is to add a new task attempt state to be able to handle a situation where a node becomes unstable while reducers are past the copy phase.

          As mentioned above, option (1) is narrower and simpler in scope, but it will ensure that the job will complete in time only after all reducers are complete. Option (2) can narrow this window further so that as soon as the we're past the copy phase we can reasonably ensure that the job will complete in time, but at the expense of a fair amount of complexity of the changes.

          FYI, our company has been running with option (1) for a while, and we've been more or less happy with that.

          I'd like to hear others' thoughts on this. The patch can be updated for the option (1) fairly quickly. It will take more time to complete option (2) on the other hand. We can discuss which option we want to go with, and even going with option (1) first in this JIRA and do option (2) in another JIRA. Thoughts?

          Show
          Sangjin Lee added a comment - Thanks for the comments Ben Podgursky . I've been meaning to revisit this JIRA. This JIRA is stalled because there are 2 options to move forward with different implications. Option (1) is to prevent re-running mappers as soon as all reducers complete, and the option (2) is to add a new task attempt state to be able to handle a situation where a node becomes unstable while reducers are past the copy phase. As mentioned above, option (1) is narrower and simpler in scope, but it will ensure that the job will complete in time only after all reducers are complete. Option (2) can narrow this window further so that as soon as the we're past the copy phase we can reasonably ensure that the job will complete in time, but at the expense of a fair amount of complexity of the changes. FYI, our company has been running with option (1) for a while, and we've been more or less happy with that. I'd like to hear others' thoughts on this. The patch can be updated for the option (1) fairly quickly. It will take more time to complete option (2) on the other hand. We can discuss which option we want to go with, and even going with option (1) first in this JIRA and do option (2) in another JIRA. Thoughts?
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 34s The applied patch generated 3 new checkstyle issues (total was 108, now 109).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 21s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 1m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 8m 52s Tests failed in hadoop-mapreduce-client-app.
              46m 0s  



          Reason Tests
          Failed unit tests hadoop.mapreduce.v2.app.TestJobEndNotifier



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12638107/mapreduce-5817.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8f73bdd
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5933/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5933/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5933/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5933/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 34s The applied patch generated 3 new checkstyle issues (total was 108, now 109). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 1m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 8m 52s Tests failed in hadoop-mapreduce-client-app.     46m 0s   Reason Tests Failed unit tests hadoop.mapreduce.v2.app.TestJobEndNotifier Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12638107/mapreduce-5817.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5933/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5933/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5933/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5933/console This message was automatically generated.
          Hide
          Ben Podgursky added a comment -

          Is this ticket on any roadmaps? We're running into a problem which I think is an extreme of this case – we have a lot of MR jobs which are Map-only, and when a NodeManager goes unhealthy while other map tasks are finishing, the Map task is re-run, even though there are no reducers at all. If the tasks are slow, this is a huge waste of time.

          Show
          Ben Podgursky added a comment - Is this ticket on any roadmaps? We're running into a problem which I think is an extreme of this case – we have a lot of MR jobs which are Map-only, and when a NodeManager goes unhealthy while other map tasks are finishing, the Map task is re-run, even though there are no reducers at all. If the tasks are slow, this is a huge waste of time.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Moving bugs out of previously closed releases into the next minor release 2.8.0.

          Show
          Vinod Kumar Vavilapalli added a comment - Moving bugs out of previously closed releases into the next minor release 2.8.0.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12638107/mapreduce-5817.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f1a152c
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5566/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12638107/mapreduce-5817.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5566/console This message was automatically generated.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12638107/mapreduce-5817.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f1a152c
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5557/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12638107/mapreduce-5817.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5557/console This message was automatically generated.
          Hide
          Gera Shegalov added a comment -

          The scope is:

          We should redefine JobImpl.checkReadyForCommit to return COMMITTING when

           
          if (numReduceTasks > 0) {
             if (isucceededReduceTaskCount == numReduceTasks) 
               return COMMITTING; 
          } else if (completedTaskCount == tasks.size()) {
             return COMMITTING; 
          }
          

          To address unprecedented nature, we can introduce a new state for TaskAttempImpl LOST_COMPLETE that only mappers can go into from SUCCEEDED on lost node.

          On successful Job commit, we should generate an event that will kill all outstanding unfinished map task attempts. Move one map task attempt form LOST to SUCCEEDED (e.g, with minimum id) if there is none yet.

          Show
          Gera Shegalov added a comment - The scope is: We should redefine JobImpl.checkReadyForCommit to return COMMITTING when if (numReduceTasks > 0) { if (isucceededReduceTaskCount == numReduceTasks) return COMMITTING; } else if (completedTaskCount == tasks.size()) { return COMMITTING; } To address unprecedented nature, we can introduce a new state for TaskAttempImpl LOST_COMPLETE that only mappers can go into from SUCCEEDED on lost node. On successful Job commit, we should generate an event that will kill all outstanding unfinished map task attempts. Move one map task attempt form LOST to SUCCEEDED (e.g, with minimum id) if there is none yet.
          Hide
          Sangjin Lee added a comment -

          Gera Shegalov I agree with the pros of option (2). On the other hand, I do feel uneasy about "resurrecting" a killed attempt which is necessary with option (2). I don't think that's done today, so it would be somewhat unprecedented. Also, what do you think the scope of changes would be?

          Show
          Sangjin Lee added a comment - Gera Shegalov I agree with the pros of option (2). On the other hand, I do feel uneasy about "resurrecting" a killed attempt which is necessary with option (2). I don't think that's done today, so it would be somewhat unprecedented. Also, what do you think the scope of changes would be?
          Hide
          Gera Shegalov added a comment -

          Thanks for working on this, Sangjin Lee. I would like to advocate for Option 2 with "resurrect" where the job moves to COMMITTING once all the output is in HDFS
          a) the job succeeds faster
          b) there is no ambiguity what mappers' output was actually consumed.

          Show
          Gera Shegalov added a comment - Thanks for working on this, Sangjin Lee . I would like to advocate for Option 2 with "resurrect" where the job moves to COMMITTING once all the output is in HDFS a) the job succeeds faster b) there is no ambiguity what mappers' output was actually consumed.
          Hide
          Sangjin Lee added a comment -

          The test failures are unrelated to this patch. They are coming from MAPREDUCE-5815.

          Show
          Sangjin Lee added a comment - The test failures are unrelated to this patch. They are coming from MAPREDUCE-5815 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12638107/mapreduce-5817.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

          org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4476//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4476//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12638107/mapreduce-5817.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4476//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4476//console This message is automatically generated.
          Hide
          Sangjin Lee added a comment -

          I am leaning towards option (1) for its simplicity and smaller impact in general. It still leaves rescheduled mappers running when all reducers complete, but I think it would be a much smaller risk than the problem we're facing.

          I'll code up a patch and submit it soon. Let me know if you have suggestions/comments.

          Show
          Sangjin Lee added a comment - I am leaning towards option (1) for its simplicity and smaller impact in general. It still leaves rescheduled mappers running when all reducers complete, but I think it would be a much smaller risk than the problem we're facing. I'll code up a patch and submit it soon. Let me know if you have suggestions/comments.
          Hide
          Sangjin Lee added a comment -

          We're talking about two options for this: (1) modify JobImpl.actOnUnusableNode() so that if all reducers are completed do not reschedule mappers, and (2) modify checkReadyForCommit() so that it transitions to COMMITTING if all reducers are completed (if reducers exist) instead of checking all tasks are completed.

          Either approach seems to have some downsides.

          For (1), the change is pretty narrow (only affects the rescheduling scenario). However, it still lets the mapper tasks that were rescheduled prior to reducer completion run. So the job may linger until those mapper tasks run to completion. And if those mapper tasks fail for any reason, it may render the job as failed (even though all reducers may have succeeded in reality).

          For (2), it would be effective and would make the job finish much more quickly. On the other hand, we'd need to do something about the mapper tasks that are running at that point. They may need to be killed. Also, if the original mapper tasks were successful, we may need to "resurrect" their status from KILLED to SUCCESSFUL to avoid confusion.

          Show
          Sangjin Lee added a comment - We're talking about two options for this: (1) modify JobImpl.actOnUnusableNode() so that if all reducers are completed do not reschedule mappers, and (2) modify checkReadyForCommit() so that it transitions to COMMITTING if all reducers are completed (if reducers exist) instead of checking all tasks are completed. Either approach seems to have some downsides. For (1), the change is pretty narrow (only affects the rescheduling scenario). However, it still lets the mapper tasks that were rescheduled prior to reducer completion run. So the job may linger until those mapper tasks run to completion. And if those mapper tasks fail for any reason, it may render the job as failed (even though all reducers may have succeeded in reality). For (2), it would be effective and would make the job finish much more quickly. On the other hand, we'd need to do something about the mapper tasks that are running at that point. They may need to be killed. Also, if the original mapper tasks were successful, we may need to "resurrect" their status from KILLED to SUCCESSFUL to avoid confusion.

            People

            • Assignee:
              Sangjin Lee
              Reporter:
              Sangjin Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development