Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.24.0, 0.23.2
    • Fix Version/s: 0.23.2
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs.
    • Target Version/s:

      Description

      TestEmptyJob fails because teh FileOutputCommitter expects a directory to be created that is not created. The FileOutputCommitter should either ignore the error or create the directory itself.

      1. MR-3982.txt
        3 kB
        Robert Joseph Evans
      2. MR-3982.txt
        1 kB
        Robert Joseph Evans

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1015 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1015/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1015 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1015/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #221 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/221/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans.
        svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048)

        Result = FAILURE
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #221 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/221/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #980 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/980/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #980 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/980/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #193 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/193/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans.
        svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048)

        Result = UNSTABLE
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #193 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/193/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048) Result = UNSTABLE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1870 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1870/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047)

        Result = ABORTED
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1870 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1870/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047) Result = ABORTED vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #675 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/675/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans.
        svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048)

        Result = ABORTED
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #675 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/675/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048) Result = ABORTED vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1936 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1936/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1936 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1936/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #658 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/658/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans.
        svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #658 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/658/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #667 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/667/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans.
        svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #667 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/667/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. svn merge --ignore-ancestry -c 1299047 ../../trunk/ (Revision 1299048) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299048 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1861 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1861/)
        MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047)

        Result = SUCCESS
        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1861 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1861/ ) MAPREDUCE-3982 . Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Contributed by Robert Joseph Evans. (Revision 1299047) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299047 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I just committed this to trunk, branch-0.23 and branch-0.23.2. Thanks Robert!

        Show
        Vinod Kumar Vavilapalli added a comment - I just committed this to trunk, branch-0.23 and branch-0.23.2. Thanks Robert!
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Looks good, +1, pushing this in.

        Show
        Vinod Kumar Vavilapalli added a comment - Looks good, +1, pushing this in.
        Hide
        Ravi Prakash added a comment -

        Aaah! Got it! Thanks Bobby! +1

        Show
        Ravi Prakash added a comment - Aaah! Got it! Thanks Bobby! +1
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12517725/MR-3982.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2025//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2025//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517725/MR-3982.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2025//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2025//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        This patch adds in a unit test that verifies that it still works even without any output.

        I started to port TestEmptyJob to the MiniMrYarnCluster, but it uses a lot of methods that are specific to the JT implementation to do some sort of synchronization test. I really don't understand all of the reasons for that test and why it is trying to force the scheduler to schedule something. I will file a separate JIRA to port it over because it is probably going to take a while for me to fully understand the test.

        Show
        Robert Joseph Evans added a comment - This patch adds in a unit test that verifies that it still works even without any output. I started to port TestEmptyJob to the MiniMrYarnCluster, but it uses a lot of methods that are specific to the JT implementation to do some sort of synchronization test. I really don't understand all of the reasons for that test and why it is trying to force the scheduler to schedule something. I will file a separate JIRA to port it over because it is probably going to take a while for me to fully understand the test.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Patch looks good. Can you please port the test TestEmptyJob to maven build and possibly MiniMRYarnCluster? Tx.

        Show
        Vinod Kumar Vavilapalli added a comment - Patch looks good. Can you please port the test TestEmptyJob to maven build and possibly MiniMRYarnCluster? Tx.
        Hide
        Robert Joseph Evans added a comment -

        That would not really work. The directory structure is the following.

        <OUTPUT>/_temporary/<ATTEMPT_ID>/...

        The problem was that the code assumed that <OUTPUT>/_temporary/<ATTEMPT_ID>/ was being created by the individual map/reduce tasks. If it was not then other code would blowup, so instead of just creating <OUTPUT>/_temporary in the setup from before we are now creating <OUTPUT>/_temporary/<ATTEMPT_ID>/

        When we clean up from a job we still want to completely remove the _temporary directory, not just <OUTPUT>/_temporary/<ATTEMPT_ID>/ or there will be an extra directory sitting around in the <OUTPUT> directory that will cause other problems.

        Show
        Robert Joseph Evans added a comment - That would not really work. The directory structure is the following. <OUTPUT>/_temporary/<ATTEMPT_ID>/... The problem was that the code assumed that <OUTPUT>/_temporary/<ATTEMPT_ID>/ was being created by the individual map/reduce tasks. If it was not then other code would blowup, so instead of just creating <OUTPUT>/_temporary in the setup from before we are now creating <OUTPUT>/_temporary/<ATTEMPT_ID>/ When we clean up from a job we still want to completely remove the _temporary directory, not just <OUTPUT>/_temporary/<ATTEMPT_ID>/ or there will be an extra directory sitting around in the <OUTPUT> directory that will cause other problems.
        Hide
        Ravi Prakash added a comment -

        Would we want to change this in cleanupJob(JobContext context) too? Would that make the getPendingJobAttemptsPath() methods useless?

        Show
        Ravi Prakash added a comment - Would we want to change this in cleanupJob(JobContext context) too? Would that make the getPendingJobAttemptsPath() methods useless?
        Hide
        Robert Joseph Evans added a comment -

        The test failures are known and covered by MAPREDUCE:3729 There are no tests added, because it is fixing a test that was failing.

        Show
        Robert Joseph Evans added a comment - The test failures are known and covered by MAPREDUCE:3729 There are no tests added, because it is fixing a test that was failing.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12517467/MR-3982.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.mapred.TestJobClientGetJob
        org.apache.hadoop.mapred.TestMRWithDistributedCache
        org.apache.hadoop.mapred.TestLocalModeWithNewApis

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2017//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2017//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517467/MR-3982.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobClientGetJob org.apache.hadoop.mapred.TestMRWithDistributedCache org.apache.hadoop.mapred.TestLocalModeWithNewApis +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2017//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2017//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        This fixes the issue by createing the directory earlier instead of relying on the mapper or reducer to do it.

        Show
        Robert Joseph Evans added a comment - This fixes the issue by createing the directory earlier instead of relying on the mapper or reducer to do it.

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development