Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6406

Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT to match mapred-default.xml

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      MAPREDUCE-6336 updated the default version for the property mapreduce.fileoutputcommitter.algorithm.version to 2. Should the FileOutputCommitter class default be updated to match?

      1. MAPREDUCE-6406.001.patch
        1 kB
        Ray Chiang
      2. MAPREDUCE-6406.002.patch
        2 kB
        Ray Chiang

        Activity

        Hide
        rchiang Ray Chiang added a comment -

        Version of patch assuming the class should be updated to match mapred-default.xml

        Show
        rchiang Ray Chiang added a comment - Version of patch assuming the class should be updated to match mapred-default.xml
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 2s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 48s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 48s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core.
            39m 42s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12740460/MAPREDUCE-6406.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 20c03c9
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5818/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5818/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5818/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 2s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 48s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core.     39m 42s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740460/MAPREDUCE-6406.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 20c03c9 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5818/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5818/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5818/console This message was automatically generated.
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks Ray Chiang for the patch.

        There is a description mismatch with the default value in mapred-default.xml for the same configuration, can you correct the same as part of this patch?

        <property>
          <name>mapreduce.fileoutputcommitter.algorithm.version</name>
          <value>2</value>
          <description>The file output committer algorithm version
          valid algorithm version number: 1 or 2
          default to 1, which is the original algorithm
        
        Show
        devaraj.k Devaraj K added a comment - Thanks Ray Chiang for the patch. There is a description mismatch with the default value in mapred-default.xml for the same configuration, can you correct the same as part of this patch? <property> <name> mapreduce.fileoutputcommitter.algorithm.version </name> <value> 2 </value> <description> The file output committer algorithm version valid algorithm version number: 1 or 2 default to 1, which is the original algorithm
        Hide
        rchiang Ray Chiang added a comment -
        • Update mapred-default.xml based on Devaraj's feedback
        Show
        rchiang Ray Chiang added a comment - Update mapred-default.xml based on Devaraj's feedback
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 1s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 34s There were no new javac warning messages.
        +1 javadoc 9m 34s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 47s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.
            39m 29s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12741054/MAPREDUCE-6406.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 445b132
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5824/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5824/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5824/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 47s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.     39m 29s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12741054/MAPREDUCE-6406.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 445b132 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5824/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5824/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5824/console This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        Thanks for the catch. I don't think I'll be able to write a program to validate the description fields...

        Show
        rchiang Ray Chiang added a comment - Thanks for the catch. I don't think I'll be able to write a program to validate the description fields...
        Hide
        rchiang Ray Chiang added a comment -

        RE: No new unit tests

        Adjusting defaults to match. No need for a unit test.

        Show
        rchiang Ray Chiang added a comment - RE: No new unit tests Adjusting defaults to match. No need for a unit test.
        Hide
        devaraj.k Devaraj K added a comment -

        +1, will commit it shortly.

        Show
        devaraj.k Devaraj K added a comment - +1, will commit it shortly.
        Hide
        devaraj.k Devaraj K added a comment -

        Committed to trunk. Thanks Ray Chiang.

        Show
        devaraj.k Devaraj K added a comment - Committed to trunk. Thanks Ray Chiang .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8052 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8052/)
        MAPREDUCE-6406. Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8052 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8052/ ) MAPREDUCE-6406 . Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #967 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/967/)
        MAPREDUCE-6406. Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #967 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/967/ ) MAPREDUCE-6406 . Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/237/)
        MAPREDUCE-6406. Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/237/ ) MAPREDUCE-6406 . Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2165/)
        MAPREDUCE-6406. Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2165/ ) MAPREDUCE-6406 . Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/226/)
        MAPREDUCE-6406. Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/226/ ) MAPREDUCE-6406 . Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        Hide
        rchiang Ray Chiang added a comment -

        Thanks for the review and the commit!

        Show
        rchiang Ray Chiang added a comment - Thanks for the review and the commit!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/235/)
        MAPREDUCE-6406. Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/235/ ) MAPREDUCE-6406 . Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2183/)
        MAPREDUCE-6406. Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2183/ ) MAPREDUCE-6406 . Update (devaraj: rev 41ae7768ebb8637c09c7f9733bcfe9dc43dbce69) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java

          People

          • Assignee:
            rchiang Ray Chiang
            Reporter:
            rchiang Ray Chiang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development