Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6478

Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob().

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In some of our test cases for MR on public cloud scenario, a very big MR job with hundreds or thousands of reducers cannot finish successfully because of Job Cleanup failures which is caused by different scale/performance impact for File System on the cloud (like AzureFS) which replacing HDFS's deletion for whole directory with REST API calls on deleting each sub-directories recursively. Even it get successfully, that could take much longer time (hours) which is not necessary and waste time/resources especially in public cloud scenario.
      In these scenarios, some failures of cleanupJob can be ignored or user choose to skip cleanupJob() completely make more sense. This is because making whole job finish successfully with side effect of wasting some user spaces is much better as user's jobs are usually comes and goes in public cloud, so have choices to tolerant some temporary files exists with get rid of big job re-run (or saving job's running time) is quite effective in time/resource cost.
      We should allow user to have this option (ignore failure or skip job cleanup stage completely) especially when user know the cleanup failure is not due to HDFS abnormal status but other FS' different performance trade-off.

      1. MAPREDUCE-6478-v1.1.patch
        4 kB
        Junping Du
      2. MAPREDUCE-6478-v1.patch
        4 kB
        Junping Du

        Issue Links

          Activity

          Hide
          djp Junping Du added a comment -

          Put a quick patch to add two configurations to allow skip cleanupJob or ignore cleanupJob failures. This is quite straightforward, so unit test is unnecessary here.

          Show
          djp Junping Du added a comment - Put a quick patch to add two configurations to allow skip cleanupJob or ignore cleanupJob failures. This is quite straightforward, so unit test is unnecessary here.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 9s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 10m 13s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 48s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 45s Tests passed in hadoop-mapreduce-client-core.
              41m 53s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12756111/MAPREDUCE-6478-v1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 559c09d
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5993/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5993/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5993/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5993/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 13s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 48s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 45s Tests passed in hadoop-mapreduce-client-core.     41m 53s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756111/MAPREDUCE-6478-v1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 559c09d whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5993/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5993/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5993/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5993/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Fix whitespace issue in v1.1. patch.

          Show
          djp Junping Du added a comment - Fix whitespace issue in v1.1. patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 54s There were no new javac warning messages.
          +1 javadoc 10m 5s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 49s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 45s Tests passed in hadoop-mapreduce-client-core.
              41m 17s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12756222/MAPREDUCE-6478-v1.1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bf2f2b4
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5994/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5994/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5994/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 54s There were no new javac warning messages. +1 javadoc 10m 5s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 49s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 45s Tests passed in hadoop-mapreduce-client-core.     41m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756222/MAPREDUCE-6478-v1.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bf2f2b4 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5994/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5994/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5994/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Junping Du, patch looks good, will wait for a few days to see if there's any opposite opinions.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Junping Du , patch looks good, will wait for a few days to see if there's any opposite opinions.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8483 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8483/)
          MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8483 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8483/ ) MAPREDUCE-6478 . Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-yarn-project/CHANGES.txt
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to trunk/branch-2, thanks Junping Du!

          Show
          leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2, thanks Junping Du !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #418 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/418/)
          MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #418 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/418/ ) MAPREDUCE-6478 . Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204) hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          Hide
          djp Junping Du added a comment -

          Thanks Wangda Tan for review and commit!

          Show
          djp Junping Du added a comment - Thanks Wangda Tan for review and commit!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #411 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/411/)
          MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #411 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/411/ ) MAPREDUCE-6478 . Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204) hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2357 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2357/)
          MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2357 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2357/ ) MAPREDUCE-6478 . Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1151 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1151/)
          MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1151 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1151/ ) MAPREDUCE-6478 . Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #392 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/392/)
          MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #392 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/392/ ) MAPREDUCE-6478 . Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2331/)
          MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2331/ ) MAPREDUCE-6478 . Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) (wangda: rev 372ad270a0d7ea5c581cd9a42b3c3cb189eca204) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              djp Junping Du
              Reporter:
              djp Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development