Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9612

DistCp worker threads are not terminated after jobs are done.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: distcp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In HADOOP-11827, a producer-consumer style thread pool was introduced to parallelize the task of listing files/directories.

      We have a use case where a distcp job is run during the commit phase of a MR2 job. However, it was found distcp does not terminate ProducerConsumer thread pools properly. Because threads are not terminated, those MR2 jobs never finish.

      In a more typical use case where distcp is run as a standalone job, those threads are terminated forcefully when the java process is terminated. So these leaked threads did not become a problem.

      1. HDFS-9612.001.patch
        17 kB
        Wei-Chiu Chuang
      2. HDFS-9612.002.patch
        5 kB
        Wei-Chiu Chuang
      3. HDFS-9612.003.patch
        7 kB
        Wei-Chiu Chuang
      4. HDFS-9612.004.patch
        7 kB
        Wei-Chiu Chuang
      5. HDFS-9612.005.patch
        7 kB
        Wei-Chiu Chuang
      6. HDFS-9612.006.patch
        7 kB
        Wei-Chiu Chuang
      7. HDFS-9612.007.patch
        8 kB
        Wei-Chiu Chuang
      8. HDFS-9612.008.patch
        8 kB
        Wei-Chiu Chuang
      9. HDFS-9612.009.patch
        8 kB
        Yongjun Zhang

        Issue Links

          Activity

          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev01:
          In this patch, I created a test case that demonstrates these worker threads are not terminated.
          There are two issues with the original implementation.

          1. ProducerConsumer.shutdown() calls Executor.shutdown(), which only disallow new tasks to be submitted, but it does not interrupt existing worker threads. It should also call Executor.shutdownNow() to interrupt them.
          2. ProducerConsumer$Worker.run() swallows interrupts.

          In this patch, I updated ProducerConsumer with a solution. I tested this code with the attached simple test case, which passed, but I have not yet made sure it's thread-safe. So more work is needed.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev01: In this patch, I created a test case that demonstrates these worker threads are not terminated. There are two issues with the original implementation. ProducerConsumer.shutdown() calls Executor.shutdown(), which only disallow new tasks to be submitted, but it does not interrupt existing worker threads. It should also call Executor.shutdownNow() to interrupt them. ProducerConsumer$Worker.run() swallows interrupts. In this patch, I updated ProducerConsumer with a solution. I tested this code with the attached simple test case, which passed, but I have not yet made sure it's thread-safe. So more work is needed.
          Hide
          drankye Kai Zheng added a comment -

          Oh, the patch is rather messy. You need to update to the latest trunk and rebase your work. The work itself looks good, though. Thanks Wei chiu!

          Show
          drankye Kai Zheng added a comment - Oh, the patch is rather messy. You need to update to the latest trunk and rebase your work. The work itself looks good, though. Thanks Wei chiu!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I am sorry for not rebasing the patch – rev02 is a patch against trunk head.

          Show
          jojochuang Wei-Chiu Chuang added a comment - I am sorry for not rebasing the patch – rev02 is a patch against trunk head.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Again, this will likely need more testing to ensure its thread-safety.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Again, this will likely need more testing to ensure its thread-safety.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev03:

          1. Added another (more complex) test case; also, make sure all ProducerConsumer tests call shutdown() to terminate threads.
          2. Simplified ProducerConsumer$Worker.run() logic. In SimpleCopyListing, ProducerConsumer.shutdown() is called after all work is consumed, so there is no need to consider the case where workers are interrupted in the middle of getting or putting or processing a work. Therefore, all workers are supposed to wait at
            work = inputQueue.take();
            

            and if it gets an interrupt, simply return;

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev03: Added another (more complex) test case; also, make sure all ProducerConsumer tests call shutdown() to terminate threads. Simplified ProducerConsumer$Worker.run() logic. In SimpleCopyListing, ProducerConsumer.shutdown() is called after all work is consumed, so there is no need to consider the case where workers are interrupted in the middle of getting or putting or processing a work. Therefore, all workers are supposed to wait at work = inputQueue.take(); and if it gets an interrupt, simply return;
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 20s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_66
          +1 compile 0m 17s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 9s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_66
          +1 javac 0m 12s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_91
          +1 javac 0m 14s the patch passed
          -1 checkstyle 0m 9s Patch generated 2 new checkstyle issues in hadoop-tools/hadoop-distcp (total was 5, now 6).
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 32s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          -1 unit 6m 43s hadoop-distcp in the patch failed with JDK v1.8.0_66.
          -1 unit 6m 43s hadoop-distcp in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          26m 37s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.tools.util.TestProducerConsumer
          JDK v1.7.0_91 Failed junit tests hadoop.tools.util.TestProducerConsumer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780574/HDFS-9612.003.patch
          JIRA Issue HDFS-9612
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9b82bb1813fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 96d8f1d
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14030/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14030/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 20s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_66 +1 compile 0m 17s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 9s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 17s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_66 +1 javac 0m 12s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_91 +1 javac 0m 14s the patch passed -1 checkstyle 0m 9s Patch generated 2 new checkstyle issues in hadoop-tools/hadoop-distcp (total was 5, now 6). +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 32s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 -1 unit 6m 43s hadoop-distcp in the patch failed with JDK v1.8.0_66. -1 unit 6m 43s hadoop-distcp in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 26m 37s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.tools.util.TestProducerConsumer JDK v1.7.0_91 Failed junit tests hadoop.tools.util.TestProducerConsumer Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780574/HDFS-9612.003.patch JIRA Issue HDFS-9612 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9b82bb1813fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 96d8f1d Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14030/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14030/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14030/console This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev04: replace commons.logging with slf4j.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev04: replace commons.logging with slf4j.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 20s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_66
          +1 compile 0m 17s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 10s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 16s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.8.0_66
          +1 javac 0m 13s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.7.0_91
          +1 javac 0m 15s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 22s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 37s the patch passed
          -1 javadoc 1m 7s hadoop-tools_hadoop-distcp-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 51, now 51).
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_91
          +1 unit 7m 52s hadoop-distcp in the patch passed with JDK v1.8.0_66.
          +1 unit 7m 36s hadoop-distcp in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          30m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780631/HDFS-9612.004.patch
          JIRA Issue HDFS-9612
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 34d7e6ab87e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 28bd138
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          javadoc hadoop-tools_hadoop-distcp-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/14037/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-distcp-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14037/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14037/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 20s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_66 +1 compile 0m 17s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 10s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 18s the patch passed +1 compile 0m 13s the patch passed with JDK v1.8.0_66 +1 javac 0m 13s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_91 +1 javac 0m 15s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 37s the patch passed -1 javadoc 1m 7s hadoop-tools_hadoop-distcp-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 51, now 51). +1 javadoc 0m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_91 +1 unit 7m 52s hadoop-distcp in the patch passed with JDK v1.8.0_66. +1 unit 7m 36s hadoop-distcp in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 30m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780631/HDFS-9612.004.patch JIRA Issue HDFS-9612 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 34d7e6ab87e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28bd138 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 javadoc hadoop-tools_hadoop-distcp-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/14037/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-distcp-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14037/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14037/console This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev05: added @throws to make Javadoc happy.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev05: added @throws to make Javadoc happy.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 42s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_66
          +1 compile 0m 18s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 8s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 17s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_66
          +1 javac 0m 12s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.7.0_91
          +1 javac 0m 15s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 32s the patch passed
          -1 javadoc 0m 57s hadoop-tools_hadoop-distcp-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 51, now 51).
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          +1 unit 6m 53s hadoop-distcp in the patch passed with JDK v1.8.0_66.
          +1 unit 6m 50s hadoop-distcp in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          27m 26s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780744/HDFS-9612.005.patch
          JIRA Issue HDFS-9612
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b36a01e84ad5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2d16f40
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          javadoc hadoop-tools_hadoop-distcp-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/14042/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-distcp-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14042/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14042/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 42s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_66 +1 compile 0m 18s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 8s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 19s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_66 +1 javac 0m 12s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_91 +1 javac 0m 15s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 32s the patch passed -1 javadoc 0m 57s hadoop-tools_hadoop-distcp-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 51, now 51). +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 +1 unit 6m 53s hadoop-distcp in the patch passed with JDK v1.8.0_66. +1 unit 6m 50s hadoop-distcp in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 27m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780744/HDFS-9612.005.patch JIRA Issue HDFS-9612 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b36a01e84ad5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2d16f40 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 javadoc hadoop-tools_hadoop-distcp-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/14042/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-distcp-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14042/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14042/console This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev06: make javadocs happy.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev06: make javadocs happy.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 35s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_66
          +1 compile 0m 17s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 9s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_66
          +1 javac 0m 11s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_91
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 8s the patch passed
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 34s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          +1 unit 6m 32s hadoop-distcp in the patch passed with JDK v1.8.0_66.
          +1 unit 6m 40s hadoop-distcp in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          26m 40s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781254/HDFS-9612.006.patch
          JIRA Issue HDFS-9612
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f5fe05bfa954 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 38c4c14
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14072/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14072/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 35s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_66 +1 compile 0m 17s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 9s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 18s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_66 +1 javac 0m 11s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_91 +1 javac 0m 14s the patch passed +1 checkstyle 0m 8s the patch passed +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 +1 unit 6m 32s hadoop-distcp in the patch passed with JDK v1.8.0_66. +1 unit 6m 40s hadoop-distcp in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 26m 40s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781254/HDFS-9612.006.patch JIRA Issue HDFS-9612 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f5fe05bfa954 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 38c4c14 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14072/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14072/console This message was automatically generated.
          Hide
          3opan Zoran Dimitrijevic added a comment -

          168:
          try

          { work = inputQueue.take(); }

          catch (InterruptedException e)

          { LOG.debug("Interrupted while waiting for request from inputQueue."); // if interrupt is triggered by shutdown(), terminate the thread // otherwise, attempt to take again Thread.currentThread().interrupt(); return; }

          I don't see how you distinguish between shutdown and other interrupts? How I read this code, you actually stop the thread using return every time thread gets any InterruptedException. Whether it is possible to get this exception for any reason except calling shutdown, I am not 100% certain - that is why I protected this thread (but apparently failed to realize shutdown is not able to complete). What you want is to let thread execute shutdown() but continue working if it is interrupted while waiting to take() from input queue.

          Give me a day to find time to check how to do this, or explain me how this return does not get out of run() method and finishes the thread. Or maybe it does work if you simply remove "return" from your patch?

          Show
          3opan Zoran Dimitrijevic added a comment - 168: try { work = inputQueue.take(); } catch (InterruptedException e) { LOG.debug("Interrupted while waiting for request from inputQueue."); // if interrupt is triggered by shutdown(), terminate the thread // otherwise, attempt to take again Thread.currentThread().interrupt(); return; } I don't see how you distinguish between shutdown and other interrupts? How I read this code, you actually stop the thread using return every time thread gets any InterruptedException. Whether it is possible to get this exception for any reason except calling shutdown, I am not 100% certain - that is why I protected this thread (but apparently failed to realize shutdown is not able to complete). What you want is to let thread execute shutdown() but continue working if it is interrupted while waiting to take() from input queue. Give me a day to find time to check how to do this, or explain me how this return does not get out of run() method and finishes the thread. Or maybe it does work if you simply remove "return" from your patch?
          Hide
          3opan Zoran Dimitrijevic added a comment -

          OK, so I read a bit more about this: http://www.ibm.com/developerworks/library/j-jtp05236/

          From what I see, the only way to get InterruptedException is by some other thread sending it - so what you do here is correct (sorry for my C++ mind not really understanding what java expects causing this issue).

          LGTM.

          Only suggestion I might have is that logging change is not related to this fix, which might make downporting this patch a bit harder. But, it is up to Hadoop committers to make that decision. Sorry for not properly shutting down these threads after they are not needed anymore.

          Show
          3opan Zoran Dimitrijevic added a comment - OK, so I read a bit more about this: http://www.ibm.com/developerworks/library/j-jtp05236/ From what I see, the only way to get InterruptedException is by some other thread sending it - so what you do here is correct (sorry for my C++ mind not really understanding what java expects causing this issue). LGTM. Only suggestion I might have is that logging change is not related to this fix, which might make downporting this patch a bit harder. But, it is up to Hadoop committers to make that decision. Sorry for not properly shutting down these threads after they are not needed anymore.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks Zoran Dimitrijevic for spending much time reviewing the code! This issue only becomes a problem to a special use case.

          I'm sorry I did not have time to respond to your comments. What do you mean by "logging change is not related to this fix"? Thanks again

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks Zoran Dimitrijevic for spending much time reviewing the code! This issue only becomes a problem to a special use case. I'm sorry I did not have time to respond to your comments. What do you mean by "logging change is not related to this fix"? Thanks again
          Hide
          3opan Zoran Dimitrijevic added a comment -

          There is a change from Log to slf4j:

          -import org.apache.commons.logging.Log;
          -import org.apache.commons.logging.LogFactory;
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;

          I don't know what is the reason for this change (but I'm sure there is a reason since distcp is an old codebase), but it is unrelated to the core change (InterruptedException change). If someone needs to cherry-pick this patch to older version of hadoop, then maybe they don't want to change logger. But, this is up to hadoop committer to decide - alternative is to make another patch which goes through all distcp .java files and changes loggers.

          Show
          3opan Zoran Dimitrijevic added a comment - There is a change from Log to slf4j: -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; I don't know what is the reason for this change (but I'm sure there is a reason since distcp is an old codebase), but it is unrelated to the core change (InterruptedException change). If someone needs to cherry-pick this patch to older version of hadoop, then maybe they don't want to change logger. But, this is up to hadoop committer to decide - alternative is to make another patch which goes through all distcp .java files and changes loggers.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks for Wei-Chiu Chuang's work here and Zoran Dimitrijevic for the review.

          Overall the patch looks good. I have the following comments:

          1. About the following code
                executor.shutdown();
                executor.shutdownNow();
            

            it looks a bit werid to me. Instead of calling two methods, why not just call {{ executor.shutdownNow();}}?

          2. Agree with Zoran that separating the log4j change to a different jira would be better.
          3. About
                    try {
                      work = inputQueue.take();
                    } catch (InterruptedException e) {
                      LOG.debug("Interrupted while waiting for request from inputQueue.");
                      // if interrupt is triggered by shutdown(), terminate the thread
                      // otherwise, attempt to take again
                      Thread.currentThread().interrupt();
                      return;
                    }
            
                    boolean isDone = false;
                    while (!isDone) {
                      try {
                        // assume processor.processItem() is stateless
                        WorkReport<R> result = processor.processItem(work);
                        outputQueue.put(result);
                        isDone = true;
                      } catch (InterruptedException ie) {
                        LOG.debug("Could not put report into outputQueue. Retrying...");
                      }
                    }
            
            1. The call to Thread.currentThread().interrupt(); can be dropped
            2. If I understand it correctly, the comment "if interrupt is triggered by shutdown(), terminate the thread; otherwise, attempt to take again" can be improved. such as "If interrupt happens when taking work out from queue, then the interrupt is likely triggered by the shutdown() call, exit the thread; if the interrupt happens while the work is being processed, go back to process the same work again."
            3. The message ""Could not put report into outputQueue" is not accurate since interrupt can be triggered from either within processItem or put operation.
            4. Add javadoc to this method and probably even the class itself to say that it assumes " processor.processItem() is stateless"
          4. About the test, would you please put some comment to indicate how the test would fail and with the fix it won't fail?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks for Wei-Chiu Chuang 's work here and Zoran Dimitrijevic for the review. Overall the patch looks good. I have the following comments: About the following code executor.shutdown(); executor.shutdownNow(); it looks a bit werid to me. Instead of calling two methods, why not just call {{ executor.shutdownNow();}}? Agree with Zoran that separating the log4j change to a different jira would be better. About try { work = inputQueue.take(); } catch (InterruptedException e) { LOG.debug( "Interrupted while waiting for request from inputQueue." ); // if interrupt is triggered by shutdown(), terminate the thread // otherwise, attempt to take again Thread .currentThread().interrupt(); return ; } boolean isDone = false ; while (!isDone) { try { // assume processor.processItem() is stateless WorkReport<R> result = processor.processItem(work); outputQueue.put(result); isDone = true ; } catch (InterruptedException ie) { LOG.debug( "Could not put report into outputQueue. Retrying..." ); } } The call to Thread.currentThread().interrupt(); can be dropped If I understand it correctly, the comment "if interrupt is triggered by shutdown(), terminate the thread; otherwise, attempt to take again" can be improved. such as "If interrupt happens when taking work out from queue, then the interrupt is likely triggered by the shutdown() call, exit the thread; if the interrupt happens while the work is being processed, go back to process the same work again." The message ""Could not put report into outputQueue" is not accurate since interrupt can be triggered from either within processItem or put operation. Add javadoc to this method and probably even the class itself to say that it assumes " processor.processItem() is stateless" About the test, would you please put some comment to indicate how the test would fail and with the fix it won't fail? Thanks.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev07:
          Thanks Yongjun Zhang for comments. I uploaded a patch that address most issues. The change of slf4j was used in conjunction of GenericTestUtils.setLogLevel to set log level to DEBUG. GenericTestUtils.setLogLevel is a useful tool, but unfortunately requires slf4j. It is not a necessary part of the fix, so I removed them.

          About the tests, they use GenericTestUtils.waitForThreadTermination() which periodically checks to see if there are any threads whose name matches the pattern "pool-.thread." (it's regular expression). These are the threads created by ExecutorService. If the fix works, those threads should terminate right away after ProducerConsumer.shutdown() is called.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev07: Thanks Yongjun Zhang for comments. I uploaded a patch that address most issues. The change of slf4j was used in conjunction of GenericTestUtils.setLogLevel to set log level to DEBUG. GenericTestUtils.setLogLevel is a useful tool, but unfortunately requires slf4j. It is not a necessary part of the fix, so I removed them. About the tests, they use GenericTestUtils.waitForThreadTermination() which periodically checks to see if there are any threads whose name matches the pattern "pool-. thread. " (it's regular expression). These are the threads created by ExecutorService. If the fix works, those threads should terminate right away after ProducerConsumer.shutdown() is called.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 42s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_66
          +1 compile 0m 18s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 9s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.8.0_66
          +1 javac 0m 13s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.7.0_91
          +1 javac 0m 15s the patch passed
          -1 checkstyle 0m 9s Patch generated 1 new checkstyle issues in hadoop-tools/hadoop-distcp (total was 2, now 3).
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 33s the patch passed
          +1 javadoc 0m 9s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_91
          +1 unit 7m 12s hadoop-distcp in the patch passed with JDK v1.8.0_66.
          +1 unit 6m 53s hadoop-distcp in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          27m 45s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782218/HDFS-9612.007.patch
          JIRA Issue HDFS-9612
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b9745a64785e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 62b0d61
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14120/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14120/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14120/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 42s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_66 +1 compile 0m 18s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 9s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 19s the patch passed +1 compile 0m 13s the patch passed with JDK v1.8.0_66 +1 javac 0m 13s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_91 +1 javac 0m 15s the patch passed -1 checkstyle 0m 9s Patch generated 1 new checkstyle issues in hadoop-tools/hadoop-distcp (total was 2, now 3). +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 33s the patch passed +1 javadoc 0m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_91 +1 unit 7m 12s hadoop-distcp in the patch passed with JDK v1.8.0_66. +1 unit 6m 53s hadoop-distcp in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 27m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782218/HDFS-9612.007.patch JIRA Issue HDFS-9612 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b9745a64785e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 62b0d61 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14120/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14120/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14120/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Wei-Chiu Chuang,

          The patch looks good to me, except a minor thing: the call to Thread.currentThread().interrupt(); in the test code does not seem necessary. +1 after addressing this comment.

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Wei-Chiu Chuang , The patch looks good to me, except a minor thing: the call to Thread.currentThread().interrupt(); in the test code does not seem necessary. +1 after addressing this comment. Thanks.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks Yongjun Zhang for the missing change. Attached is my rev8 patch.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks Yongjun Zhang for the missing change. Attached is my rev8 patch.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Wei-Chiu, +1 on rev8 pending jenkins.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Wei-Chiu, +1 on rev8 pending jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 47s trunk passed
          +1 compile 0m 16s trunk passed with JDK v1.8.0_66
          +1 compile 0m 17s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 10s trunk passed
          +1 mvnsite 0m 25s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 16s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.8.0_66
          +1 javac 0m 13s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.7.0_91
          +1 javac 0m 16s the patch passed
          -1 checkstyle 0m 10s Patch generated 1 new checkstyle issues in hadoop-tools/hadoop-distcp (total was 2, now 3).
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 35s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 15s the patch passed with JDK v1.7.0_91
          +1 unit 7m 30s hadoop-distcp in the patch passed with JDK v1.8.0_66.
          +1 unit 7m 31s hadoop-distcp in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          30m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782386/HDFS-9612.008.patch
          JIRA Issue HDFS-9612
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e40f8a9cfc31 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cdf8895
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14129/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14129/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14129/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 47s trunk passed +1 compile 0m 16s trunk passed with JDK v1.8.0_66 +1 compile 0m 17s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 10s trunk passed +1 mvnsite 0m 25s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 19s the patch passed +1 compile 0m 13s the patch passed with JDK v1.8.0_66 +1 javac 0m 13s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_91 +1 javac 0m 16s the patch passed -1 checkstyle 0m 10s Patch generated 1 new checkstyle issues in hadoop-tools/hadoop-distcp (total was 2, now 3). +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 35s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_91 +1 unit 7m 30s hadoop-distcp in the patch passed with JDK v1.8.0_66. +1 unit 7m 31s hadoop-distcp in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 30m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782386/HDFS-9612.008.patch JIRA Issue HDFS-9612 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e40f8a9cfc31 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cdf8895 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14129/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14129/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14129/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Upload rev9 on behalf of Wei-Chiu to fix checkstyle issue.

          Show
          yzhangal Yongjun Zhang added a comment - Upload rev9 on behalf of Wei-Chiu to fix checkstyle issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 26s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_66
          +1 compile 0m 17s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 9s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_66
          +1 javac 0m 12s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.7.0_91
          +1 javac 0m 15s the patch passed
          +1 checkstyle 0m 8s the patch passed
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 33s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          +1 unit 7m 13s hadoop-distcp in the patch passed with JDK v1.8.0_66.
          +1 unit 6m 32s hadoop-distcp in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          27m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782466/HDFS-9612.009.patch
          JIRA Issue HDFS-9612
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 989ce731ecb0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1da762c
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14131/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14131/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 26s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_66 +1 compile 0m 17s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 9s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 18s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_66 +1 javac 0m 12s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_91 +1 javac 0m 15s the patch passed +1 checkstyle 0m 8s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 33s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 +1 unit 7m 13s hadoop-distcp in the patch passed with JDK v1.8.0_66. +1 unit 6m 32s hadoop-distcp in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 27m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782466/HDFS-9612.009.patch JIRA Issue HDFS-9612 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 989ce731ecb0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1da762c Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14131/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14131/console This message was automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9121 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9121/)
          HDFS-9612. DistCp worker threads are not terminated after jobs are done. (yzhang: rev a9c69ebeb707801071db3cc22bfcd14f87be443a)

          • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/WorkRequestProcessor.java
          • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestProducerConsumer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ProducerConsumer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9121 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9121/ ) HDFS-9612 . DistCp worker threads are not terminated after jobs are done. (yzhang: rev a9c69ebeb707801071db3cc22bfcd14f87be443a) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/WorkRequestProcessor.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestProducerConsumer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ProducerConsumer.java
          Hide
          yzhangal Yongjun Zhang added a comment -

          Committed to trunk, branch-2, branch2.8.

          Thanks Wei-Chiu Chuang for the contribution, Kai Zheng, Zoran Dimitrijevic for the review.

          Show
          yzhangal Yongjun Zhang added a comment - Committed to trunk, branch-2, branch2.8. Thanks Wei-Chiu Chuang for the contribution, Kai Zheng , Zoran Dimitrijevic for the review.

            People

            • Assignee:
              jojochuang Wei-Chiu Chuang
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development