Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6607

Enable regex pattern matching when mapreduce.task.files.preserve.filepattern is set

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Before this fix, the files in .staging directory are always preserved when mapreduce.task.files.preserve.filepattern is set. After this fix, the files in .staging directory are preserved if the name of the directory matches the regex pattern specified by mapreduce.task.files.preserve.filepattern.
      Show
      Before this fix, the files in .staging directory are always preserved when mapreduce.task.files.preserve.filepattern is set. After this fix, the files in .staging directory are preserved if the name of the directory matches the regex pattern specified by mapreduce.task.files.preserve.filepattern.

      Description

      if either of the following configs are set, then .staging dir is not cleaned up:

      • mapreduce.task.files.preserve.failedtask
      • mapreduce.task.files.preserve.filepattern

      The former was supposed to keep only .staging of failed tasks and the latter was supposed to be used only if that task name matches against the specified regular expression.

        protected boolean keepJobFiles(JobConf conf) {
          return (conf.getKeepTaskFilesPattern() != null || conf
              .getKeepFailedTaskFiles());
        }
      
        public void cleanupStagingDir() throws IOException {
          /* make sure we clean the staging files */
          String jobTempDir = null;
          FileSystem fs = getFileSystem(getConfig());
          try {
            if (!keepJobFiles(new JobConf(getConfig()))) {
              jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
              if (jobTempDir == null) {
                LOG.warn("Job Staging directory is null");
                return;
              }
              Path jobTempDirPath = new Path(jobTempDir);
              LOG.info("Deleting staging directory " + FileSystem.getDefaultUri(getConfig()) +
                  " " + jobTempDir);
              fs.delete(jobTempDirPath, true);
            }
          } catch(IOException io) {
            LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
          }
        }
      
      1. MAPREDUCE-6607.01.patch
        7 kB
        Kai Sasaki
      2. MAPREDUCE-6607.02.patch
        7 kB
        Kai Sasaki
      3. MAPREDUCE-6607.03.patch
        8 kB
        Kai Sasaki
      4. MAPREDUCE-6607.04.patch
        8 kB
        Kai Sasaki
      5. MAPREDUCE-6607.05.patch
        10 kB
        Kai Sasaki
      6. MAPREDUCE-6607.06.patch
        10 kB
        Kai Sasaki
      7. MAPREDUCE-6607-branch-2.01.patch
        10 kB
        Kai Sasaki
      8. MAPREDUCE-6607-branch-2.02.patch
        10 kB
        Kai Sasaki

        Activity

        Hide
        maysamyabandeh Maysam Yabandeh added a comment -

        Thank you Kai Sasaki and Akira Ajisaka.

        Show
        maysamyabandeh Maysam Yabandeh added a comment - Thank you Kai Sasaki and Akira Ajisaka .
        Hide
        lewuathe Kai Sasaki added a comment -

        Akira Ajisaka Thank you so much!

        Show
        lewuathe Kai Sasaki added a comment - Akira Ajisaka Thank you so much!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9839/)
        MAPREDUCE-6607. Enable regex pattern matching when (aajisaka: rev 3b3b63081b39270ba363eb4558c5fb37fd5172d5)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9839/ ) MAPREDUCE-6607 . Enable regex pattern matching when (aajisaka: rev 3b3b63081b39270ba363eb4558c5fb37fd5172d5) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed to trunk/branch-2/branch-2.8. Thanks Kai Sasaki for the contribution.

        Show
        ajisakaa Akira Ajisaka added a comment - Committed to trunk/branch-2/branch-2.8. Thanks Kai Sasaki for the contribution.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1, the license warning is not related.

        Show
        ajisakaa Akira Ajisaka added a comment - +1, the license warning is not related.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 1s branch-2 passed
        +1 compile 0m 20s branch-2 passed with JDK v1.8.0_91
        +1 compile 0m 22s branch-2 passed with JDK v1.7.0_101
        +1 checkstyle 0m 25s branch-2 passed
        +1 mvnsite 0m 27s branch-2 passed
        +1 mvneclipse 0m 22s branch-2 passed
        +1 findbugs 0m 53s branch-2 passed
        +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_91
        +1 javadoc 0m 17s branch-2 passed with JDK v1.7.0_101
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 15s the patch passed with JDK v1.8.0_91
        +1 javac 0m 15s the patch passed
        +1 compile 0m 19s the patch passed with JDK v1.7.0_101
        +1 javac 0m 19s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 24s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 51s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_91
        +1 javadoc 0m 15s the patch passed with JDK v1.7.0_101
        +1 unit 8m 51s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_91.
        +1 unit 9m 13s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_101.
        -1 asflicense 0m 18s Patch generated 1 ASF License warnings.
        33m 13s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:babe025
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805356/MAPREDUCE-6607-branch-2.02.patch
        JIRA Issue MAPREDUCE-6607
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0180c872b8bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / e7b56cf
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6522/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6522/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6522/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 1s branch-2 passed +1 compile 0m 20s branch-2 passed with JDK v1.8.0_91 +1 compile 0m 22s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 25s branch-2 passed +1 mvnsite 0m 27s branch-2 passed +1 mvneclipse 0m 22s branch-2 passed +1 findbugs 0m 53s branch-2 passed +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_91 +1 javadoc 0m 17s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 22s the patch passed +1 compile 0m 15s the patch passed with JDK v1.8.0_91 +1 javac 0m 15s the patch passed +1 compile 0m 19s the patch passed with JDK v1.7.0_101 +1 javac 0m 19s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_101 +1 unit 8m 51s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_91. +1 unit 9m 13s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_101. -1 asflicense 0m 18s Patch generated 1 ASF License warnings. 33m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805356/MAPREDUCE-6607-branch-2.02.patch JIRA Issue MAPREDUCE-6607 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0180c872b8bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e7b56cf Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6522/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6522/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6522/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 4s branch-2 passed
        +1 compile 0m 18s branch-2 passed with JDK v1.8.0_91
        +1 compile 0m 22s branch-2 passed with JDK v1.7.0_101
        +1 checkstyle 0m 26s branch-2 passed
        +1 mvnsite 0m 30s branch-2 passed
        +1 mvneclipse 0m 19s branch-2 passed
        +1 findbugs 0m 55s branch-2 passed
        +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_91
        +1 javadoc 0m 16s branch-2 passed with JDK v1.7.0_101
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.8.0_91
        +1 javac 0m 16s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.7.0_101
        +1 javac 0m 20s the patch passed
        -1 checkstyle 0m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 4 new + 323 unchanged - 0 fixed = 327 total (was 323)
        +1 mvnsite 0m 24s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 53s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_91
        +1 javadoc 0m 14s the patch passed with JDK v1.7.0_101
        +1 unit 8m 44s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_91.
        +1 unit 9m 7s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_101.
        -1 asflicense 0m 19s Patch generated 1 ASF License warnings.
        35m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:babe025
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805175/MAPREDUCE-6607-branch-2.01.patch
        JIRA Issue MAPREDUCE-6607
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fc71705ecfa9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / eb16056
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6520/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6520/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6520/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6520/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 4s branch-2 passed +1 compile 0m 18s branch-2 passed with JDK v1.8.0_91 +1 compile 0m 22s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 26s branch-2 passed +1 mvnsite 0m 30s branch-2 passed +1 mvneclipse 0m 19s branch-2 passed +1 findbugs 0m 55s branch-2 passed +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_91 +1 javadoc 0m 16s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 22s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_91 +1 javac 0m 16s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_101 +1 javac 0m 20s the patch passed -1 checkstyle 0m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 4 new + 323 unchanged - 0 fixed = 327 total (was 323) +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_101 +1 unit 8m 44s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_91. +1 unit 9m 7s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_101. -1 asflicense 0m 19s Patch generated 1 ASF License warnings. 35m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805175/MAPREDUCE-6607-branch-2.01.patch JIRA Issue MAPREDUCE-6607 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fc71705ecfa9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / eb16056 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6520/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6520/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6520/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6520/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 2m 54s Docker failed to build yetus/hadoop:babe025.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805175/MAPREDUCE-6607-branch-2.01.patch
        JIRA Issue MAPREDUCE-6607
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6518/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 2m 54s Docker failed to build yetus/hadoop:babe025. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805175/MAPREDUCE-6607-branch-2.01.patch JIRA Issue MAPREDUCE-6607 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6518/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Akira Ajisaka Thanks for taking care, I created the patch against branch-2. Could you check that?

        Show
        lewuathe Kai Sasaki added a comment - Akira Ajisaka Thanks for taking care, I created the patch against branch-2. Could you check that?
        Hide
        ajisakaa Akira Ajisaka added a comment - - edited

        Hi Kai Sasaki, would you provide a patch for branch-2? The patch cannot be applied to branch-2.

        Show
        ajisakaa Akira Ajisaka added a comment - - edited Hi Kai Sasaki , would you provide a patch for branch-2? The patch cannot be applied to branch-2.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1, sorry for delay. Committing this.

        Show
        ajisakaa Akira Ajisaka added a comment - +1, sorry for delay. Committing this.
        Hide
        lewuathe Kai Sasaki added a comment -

        Akira Ajisaka I know you are busy and sorry for bothering, but could you review this when you get a chance?

        Show
        lewuathe Kai Sasaki added a comment - Akira Ajisaka I know you are busy and sorry for bothering, but could you review this when you get a chance?
        Hide
        lewuathe Kai Sasaki added a comment -

        Fixed style issue.

        Show
        lewuathe Kai Sasaki added a comment - Fixed style issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 36s trunk passed
        +1 compile 0m 25s trunk passed with JDK v1.8.0_77
        +1 compile 0m 24s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 52s trunk passed
        +1 javadoc 0m 16s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 22s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 21s the patch passed with JDK v1.8.0_77
        +1 javac 0m 21s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_95
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 57s the patch passed
        +1 javadoc 0m 11s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
        +1 unit 9m 26s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_77.
        +1 unit 10m 7s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        35m 43s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798909/MAPREDUCE-6607.06.patch
        JIRA Issue MAPREDUCE-6607
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8e89fd6a0160 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6e6b6dd
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6433/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6433/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 36s trunk passed +1 compile 0m 25s trunk passed with JDK v1.8.0_77 +1 compile 0m 24s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 52s trunk passed +1 javadoc 0m 16s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 22s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 28s the patch passed +1 compile 0m 21s the patch passed with JDK v1.8.0_77 +1 javac 0m 21s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_95 +1 javac 0m 23s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 57s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 9m 26s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_77. +1 unit 10m 7s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 35m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798909/MAPREDUCE-6607.06.patch JIRA Issue MAPREDUCE-6607 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8e89fd6a0160 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6e6b6dd Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6433/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6433/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 9s trunk passed
        +1 compile 0m 20s trunk passed with JDK v1.8.0_77
        +1 compile 0m 22s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 46s trunk passed
        +1 javadoc 0m 15s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.8.0_77
        +1 javac 0m 16s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.7.0_95
        +1 javac 0m 20s the patch passed
        -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 1 new + 71 unchanged - 0 fixed = 72 total (was 71)
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 57s the patch passed
        +1 javadoc 0m 14s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95
        +1 unit 9m 40s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_77.
        +1 unit 9m 55s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        34m 23s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798473/MAPREDUCE-6607.05.patch
        JIRA Issue MAPREDUCE-6607
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2e40e7adc04d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 35f0770
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6429/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6429/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6429/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 9s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_77 +1 compile 0m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 46s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 22s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_77 +1 javac 0m 16s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 1 new + 71 unchanged - 0 fixed = 72 total (was 71) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 57s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95 +1 unit 9m 40s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_77. +1 unit 9m 55s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 34m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798473/MAPREDUCE-6607.05.patch JIRA Issue MAPREDUCE-6607 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2e40e7adc04d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 35f0770 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6429/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6429/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6429/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Akira Ajisaka Thank you so much for taking care. I updated the patch. Could you check it when you get a chance?

        Show
        lewuathe Kai Sasaki added a comment - Akira Ajisaka Thank you so much for taking care. I updated the patch. Could you check it when you get a chance?
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Additional comment: Would you add a test case to verify that the staging files that are not matched to file pattern are not preserved?

        Show
        ajisakaa Akira Ajisaka added a comment - Additional comment: Would you add a test case to verify that the staging files that are not matched to file pattern are not preserved?
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Three comments from me:
        1. Would you add null check to jobTempDir before Paths.get(jobTempDir).getFileName().toString()?
        2. The method name "isTaskFailed" looks misunderstanding. Would you rename the method to "isKeepFailedTaskFiles" or something?
        3. In TestStagingCleanup.java, would you unset the following setting in @After method?

            conf.setBoolean(MRJobConfig.PRESERVE_FAILED_TASK_FILES, true);
        
        Show
        ajisakaa Akira Ajisaka added a comment - Three comments from me: 1. Would you add null check to jobTempDir before Paths.get(jobTempDir).getFileName().toString() ? 2. The method name "isTaskFailed" looks misunderstanding. Would you rename the method to "isKeepFailedTaskFiles" or something? 3. In TestStagingCleanup.java, would you unset the following setting in @After method? conf.setBoolean(MRJobConfig.PRESERVE_FAILED_TASK_FILES, true );
        Hide
        lewuathe Kai Sasaki added a comment -

        Tsuyoshi Ozawa I rebased the patch on trunk. Could you check it again please?

        Show
        lewuathe Kai Sasaki added a comment - Tsuyoshi Ozawa I rebased the patch on trunk. Could you check it again please?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 33s trunk passed
        +1 compile 0m 18s trunk passed with JDK v1.8.0_74
        +1 compile 0m 22s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 42s trunk passed
        +1 javadoc 0m 14s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.8.0_74
        +1 javac 0m 16s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.7.0_95
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 53s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
        +1 unit 9m 18s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_74.
        +1 unit 9m 54s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        33m 17s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796790/MAPREDUCE-6607.04.patch
        JIRA Issue MAPREDUCE-6607
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0665a53a5659 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1e6f929
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6415/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6415/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 33s trunk passed +1 compile 0m 18s trunk passed with JDK v1.8.0_74 +1 compile 0m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 42s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_74 +1 javac 0m 16s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 9m 18s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_74. +1 unit 9m 54s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 33m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796790/MAPREDUCE-6607.04.patch JIRA Issue MAPREDUCE-6607 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0665a53a5659 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1e6f929 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6415/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6415/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Tsuyoshi Ozawa Sorry for bothering but could you review it again please?

        Show
        lewuathe Kai Sasaki added a comment - Tsuyoshi Ozawa Sorry for bothering but could you review it again please?
        Hide
        lewuathe Kai Sasaki added a comment -

        Tsuyoshi Ozawa Sorry for late response. I updated the patch. Could you check again please?

        Show
        lewuathe Kai Sasaki added a comment - Tsuyoshi Ozawa Sorry for late response. I updated the patch. Could you check again please?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 48s trunk passed
        +1 compile 0m 21s trunk passed with JDK v1.8.0_72
        +1 compile 0m 23s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 44s trunk passed
        +1 javadoc 0m 15s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 18s the patch passed with JDK v1.8.0_72
        +1 javac 0m 18s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.7.0_95
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 54s the patch passed
        +1 javadoc 0m 13s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95
        +1 unit 9m 27s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_72.
        -1 unit 10m 4s hadoop-mapreduce-client-app in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        34m 11s



        Reason Tests
        JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestJobImpl



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788995/MAPREDUCE-6607.03.patch
        JIRA Issue MAPREDUCE-6607
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3efb9d009bbf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5e7d4d5
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6336/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6336/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6336/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6336/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 48s trunk passed +1 compile 0m 21s trunk passed with JDK v1.8.0_72 +1 compile 0m 23s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 44s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 18s the patch passed with JDK v1.8.0_72 +1 javac 0m 18s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 54s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95 +1 unit 9m 27s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_72. -1 unit 10m 4s hadoop-mapreduce-client-app in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 34m 11s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestJobImpl Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788995/MAPREDUCE-6607.03.patch JIRA Issue MAPREDUCE-6607 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3efb9d009bbf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5e7d4d5 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6336/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6336/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6336/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6336/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ozawa Tsuyoshi Ozawa added a comment -

        Kai Sasaki thank you for updating. How about adding tests when both PRESERVE_FAILED_TASK_FILES and PRESERVE_FILES_PATTERN are true?

        Show
        ozawa Tsuyoshi Ozawa added a comment - Kai Sasaki thank you for updating. How about adding tests when both PRESERVE_FAILED_TASK_FILES and PRESERVE_FILES_PATTERN are true?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 14m 2s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 40s trunk passed
        +1 compile 0m 18s trunk passed with JDK v1.8.0_72
        +1 compile 0m 23s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 44s trunk passed
        +1 javadoc 0m 14s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 18s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.8.0_72
        +1 javac 0m 16s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.7.0_95
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 51s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
        +1 unit 9m 8s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_72.
        +1 unit 9m 46s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        46m 52s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787846/MAPREDUCE-6607.02.patch
        JIRA Issue MAPREDUCE-6607
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 9b397e8ccc27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ec12ce8
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6322/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6322/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 14m 2s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 40s trunk passed +1 compile 0m 18s trunk passed with JDK v1.8.0_72 +1 compile 0m 23s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 44s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 18s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_72 +1 javac 0m 16s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 9m 8s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_72. +1 unit 9m 46s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 46m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787846/MAPREDUCE-6607.02.patch JIRA Issue MAPREDUCE-6607 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9b397e8ccc27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ec12ce8 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6322/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6322/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Akira Ajisaka Thank you so much for reviewing.
        I updated the patch to keep the logic of cleaning failed task as it is now. And I only applied file pattern match for log directory files.

        Show
        lewuathe Kai Sasaki added a comment - Akira Ajisaka Thank you so much for reviewing. I updated the patch to keep the logic of cleaning failed task as it is now. And I only applied file pattern match for log directory files.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Hi Maysam Yabandeh and Kai Sasaki, I'm thinking it is reasonable that .staging dir is not cleaned if either of the two parameters is set. This is because there may be some failed tasks even if the mapreduce job is succeeded.

        The former was supposed to keep only .staging of failed tasks

        AFAIK, the files in .staging can be used for all tasks, so I'm thinking it's difficult to search what is the .staging of the failed tasks.

        By the way, now regex match is not done even if the "mapreduce.task.files.preserve.filepattern" is set. We need to fix it.

        Show
        ajisakaa Akira Ajisaka added a comment - Hi Maysam Yabandeh and Kai Sasaki , I'm thinking it is reasonable that .staging dir is not cleaned if either of the two parameters is set. This is because there may be some failed tasks even if the mapreduce job is succeeded. The former was supposed to keep only .staging of failed tasks AFAIK, the files in .staging can be used for all tasks, so I'm thinking it's difficult to search what is the .staging of the failed tasks. By the way, now regex match is not done even if the "mapreduce.task.files.preserve.filepattern" is set. We need to fix it.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Fixed. Thanks Kai Sasaki for reporting this.

        Show
        ajisakaa Akira Ajisaka added a comment - Fixed. Thanks Kai Sasaki for reporting this.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Is it included by accident?

        Yes. I'm sorry about that. I'll fix it shortly.

        Show
        ajisakaa Akira Ajisaka added a comment - Is it included by accident? Yes. I'm sorry about that. I'll fix it shortly.
        Hide
        lewuathe Kai Sasaki added a comment -

        Tsuyoshi Ozawa Sure, but it's weird. Although HDFS-9686 patch does not seem to include the test of TestStagingCleanup, the commit includes the test code I've written here.

        git show fe124da5ffc16e4795c3dd5542accd58361e1b08
        ...
        --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
        +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
        @@ -245,6 +245,63 @@ public void testDeletionofStagingOnKillLastTry() throws IOException {
              verify(fs).delete(stagingJobPath, true);
            }
        
        +  @Test
        +  public void testByPreserveFailedStaging() throws IOException {
        +    conf.set(MRJobConfig.MAPREDUCE_JOB_DIR, stagingJobDir);
        +    // Failed task's staging files should be kept
        +    conf.setBoolean(MRJobConfig.PRESERVE_FAILED_TASK_FILES, true);
        +    fs = mock(FileSystem.class);
        +    when(fs.delete(any(Path.class), anyBoolean())).thenReturn(true);
        ...
        

        Updating TestStagingCleanup does not seem to related to HDFS-9686 basically. Is it included by accident?

        Show
        lewuathe Kai Sasaki added a comment - Tsuyoshi Ozawa Sure, but it's weird. Although HDFS-9686 patch does not seem to include the test of TestStagingCleanup , the commit includes the test code I've written here. git show fe124da5ffc16e4795c3dd5542accd58361e1b08 ... --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java @@ -245,6 +245,63 @@ public void testDeletionofStagingOnKillLastTry() throws IOException { verify(fs).delete(stagingJobPath, true ); } + @Test + public void testByPreserveFailedStaging() throws IOException { + conf.set(MRJobConfig.MAPREDUCE_JOB_DIR, stagingJobDir); + // Failed task's staging files should be kept + conf.setBoolean(MRJobConfig.PRESERVE_FAILED_TASK_FILES, true ); + fs = mock(FileSystem.class); + when(fs.delete(any(Path.class), anyBoolean())).thenReturn( true ); ... Updating TestStagingCleanup does not seem to related to HDFS-9686 basically. Is it included by accident?
        Hide
        ozawa Tsuyoshi Ozawa added a comment -

        Kai Sasaki the patch seems to be stale, so could you rebase it on trunk?

        Show
        ozawa Tsuyoshi Ozawa added a comment - Kai Sasaki the patch seems to be stale, so could you rebase it on trunk?
        Hide
        ozawa Tsuyoshi Ozawa added a comment -

        OK, I'll check this issue today.

        Show
        ozawa Tsuyoshi Ozawa added a comment - OK, I'll check this issue today.
        Hide
        lewuathe Kai Sasaki added a comment -

        Tsuyoshi Ozawa Sorry for bothering you but could you review this?

        Show
        lewuathe Kai Sasaki added a comment - Tsuyoshi Ozawa Sorry for bothering you but could you review this?
        Hide
        lewuathe Kai Sasaki added a comment -

        Could someone review this?

        Show
        lewuathe Kai Sasaki added a comment - Could someone review this?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 35s trunk passed
        +1 compile 0m 20s trunk passed with JDK v1.8.0_66
        +1 compile 0m 22s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 43s trunk passed
        +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 18s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.8.0_66
        +1 javac 0m 16s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.7.0_91
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 48s the patch passed
        +1 javadoc 0m 13s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 16s the patch passed with JDK v1.7.0_91
        +1 unit 9m 12s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_66.
        +1 unit 9m 49s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        33m 54s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782477/MAPREDUCE-6607.01.patch
        JIRA Issue MAPREDUCE-6607
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a5dcf0affe81 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1da762c
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6271/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6271/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 35s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_66 +1 compile 0m 22s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 43s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 18s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 22s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_66 +1 javac 0m 16s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_91 +1 javac 0m 20s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 48s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 16s the patch passed with JDK v1.7.0_91 +1 unit 9m 12s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_66. +1 unit 9m 49s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 33m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782477/MAPREDUCE-6607.01.patch JIRA Issue MAPREDUCE-6607 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a5dcf0affe81 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1da762c Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6271/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6271/console This message was automatically generated.

          People

          • Assignee:
            lewuathe Kai Sasaki
            Reporter:
            maysamyabandeh Maysam Yabandeh
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development