Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10682

Replace FsDatasetImpl object lock with a separate lock object

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This Jira proposes to replace the FsDatasetImpl object lock with a separate lock object. Doing so will make it easier to measure lock statistics like lock held time and warn about potential lock contention due to slow disk operations.

      Right now we can use org.apache.hadoop.util.AutoCloseableLock. In the future we can also consider replacing the lock with a read-write lock.

      1. HDFS-10682.001.patch
        115 kB
        Chen Liang
      2. HDFS-10682.002.patch
        114 kB
        Chen Liang
      3. HDFS-10682.003.patch
        114 kB
        Chen Liang
      4. HDFS-10682.004.patch
        115 kB
        Chen Liang
      5. HDFS-10682.005.patch
        115 kB
        Chen Liang
      6. HDFS-10682.006.patch
        63 kB
        Chen Liang
      7. HDFS-10682.007.patch
        70 kB
        Chen Liang
      8. HDFS-10682.008.patch
        70 kB
        Chen Liang
      9. HDFS-10682.009.patch
        69 kB
        Chen Liang
      10. HDFS-10682.010.patch
        68 kB
        Chen Liang
      11. HDFS-10682-branch-2.001.patch
        64 kB
        Chen Liang
      12. HDFS-10682-branch-2.002.patch
        64 kB
        Chen Liang
      13. HDFS-10682-branch-2.003.patch
        64 kB
        Chen Liang
      14. HDFS-10682-branch-2.004.patch
        64 kB
        Chen Liang
      15. HDFS-10682-branch-2.005.patch
        66 kB
        Chen Liang
      16. HDFS-10682-branch-2.006.patch
        67 kB
        Chen Liang

        Issue Links

          Activity

          Hide
          vagarychen Chen Liang added a comment -

          added a initial patch, which includes a lock checker class exposing method for lock held time, and instrumented FsDatasetImpl to call the method.

          Show
          vagarychen Chen Liang added a comment - added a initial patch, which includes a lock checker class exposing method for lock held time, and instrumented FsDatasetImpl to call the method.
          Hide
          vagarychen Chen Liang added a comment -

          Removed an unnecessary instrumentation call

          Show
          vagarychen Chen Liang added a comment - Removed an unnecessary instrumentation call
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 4s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 11m 54s trunk passed
          +1 compile 0m 50s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 58s trunk passed
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 188 new + 79 unchanged - 41 fixed = 267 total (was 120)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 55s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 53s the patch passed
          -1 unit 61m 1s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          99m 17s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Inconsistent synchronization of org.apache.hadoop.hdfs.LockChecker.warningsSuppressed; locked 50% of time Unsynchronized access at LockChecker.java:50% of time Unsynchronized access at LockChecker.java:[line 150]
            Should org.apache.hadoop.hdfs.LockChecker$OperationLockInfomation be a static inner class? At LockChecker.java:inner class? At LockChecker.java:[lines 67-82]
          Failed junit tests hadoop.hdfs.TestParallelShortCircuitReadUnCached
            hadoop.hdfs.TestErasureCodeBenchmarkThroughput
            hadoop.hdfs.TestParallelUnixDomainRead
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820027/HDFS-10682.002.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4d097e8c7a30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 703fdf8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16173/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16173/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16173/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16173/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16173/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 4s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 11m 54s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 188 new + 79 unchanged - 41 fixed = 267 total (was 120) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 55s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 53s the patch passed -1 unit 61m 1s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 99m 17s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Inconsistent synchronization of org.apache.hadoop.hdfs.LockChecker.warningsSuppressed; locked 50% of time Unsynchronized access at LockChecker.java:50% of time Unsynchronized access at LockChecker.java: [line 150]   Should org.apache.hadoop.hdfs.LockChecker$OperationLockInfomation be a static inner class? At LockChecker.java:inner class? At LockChecker.java: [lines 67-82] Failed junit tests hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.TestParallelUnixDomainRead   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820027/HDFS-10682.002.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4d097e8c7a30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 703fdf8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16173/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16173/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/16173/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16173/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16173/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 11m 26s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 0m 58s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 69 new + 79 unchanged - 41 fixed = 148 total (was 120)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 53s the patch passed
          +1 unit 61m 14s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          85m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820070/HDFS-10682.003.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a1a445642174 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d383bfd
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16180/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16180/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16180/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 11m 26s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 69 new + 79 unchanged - 41 fixed = 148 total (was 120) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 61m 14s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 85m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820070/HDFS-10682.003.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a1a445642174 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d383bfd Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16180/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16180/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16180/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 57s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 20 new + 79 unchanged - 41 fixed = 99 total (was 120)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 62m 17s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          81m 27s



          Reason Tests
          Failed junit tests hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820220/HDFS-10682.004.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ef11cdb06df4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / da6adf5
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16191/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16191/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16191/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16191/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 57s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 20 new + 79 unchanged - 41 fixed = 99 total (was 120) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 62m 17s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 81m 27s Reason Tests Failed junit tests hadoop.cli.TestHDFSCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820220/HDFS-10682.004.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ef11cdb06df4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / da6adf5 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16191/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16191/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16191/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16191/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 4s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 51s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 79 unchanged - 41 fixed = 79 total (was 120)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 55s the patch passed
          +1 javadoc 0m 57s the patch passed
          -1 unit 63m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          83m 8s



          Reason Tests
          Failed junit tests hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820262/HDFS-10682.005.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5c72fb2f0d46 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d2cf8b5
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16197/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16197/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16197/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 4s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 79 unchanged - 41 fixed = 79 total (was 120) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 55s the patch passed +1 javadoc 0m 57s the patch passed -1 unit 63m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 83m 8s Reason Tests Failed junit tests hadoop.cli.TestHDFSCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820262/HDFS-10682.005.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5c72fb2f0d46 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d2cf8b5 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16197/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16197/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16197/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vagarychen Chen Liang added a comment -

          The failed test is HDFSCLI, which I don't think has anything to do this patch. And I did not see failure when doing this test locally.

          Show
          vagarychen Chen Liang added a comment - The failed test is HDFSCLI, which I don't think has anything to do this patch. And I did not see failure when doing this test locally.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Chen Liang, thanks for taking this up. I recommend splitting the work into two parts:

          1. Refactor the code to synchronize on a new Reentrant lock instead of the FsDatasetImpl object. (create a separate Jira for this). The advantage of a wrapper object for the lock is callers won't need to add boilerplate code for instrumentation. Also we can use try-with-resources instead of having to release the lock manually.
          2. In the second patch we can add instrumentation in just the acquire/close methods and expose it as a metric.
          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Chen Liang , thanks for taking this up. I recommend splitting the work into two parts: Refactor the code to synchronize on a new Reentrant lock instead of the FsDatasetImpl object. (create a separate Jira for this). The advantage of a wrapper object for the lock is callers won't need to add boilerplate code for instrumentation. Also we can use try-with-resources instead of having to release the lock manually. In the second patch we can add instrumentation in just the acquire/close methods and expose it as a metric.
          Hide
          vagarychen Chen Liang added a comment -

          Thanks Arpit Agarwal for the comments! Will upload another patch fixing this soon.

          Show
          vagarychen Chen Liang added a comment - Thanks Arpit Agarwal for the comments! Will upload another patch fixing this soon.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the updated patch Chen Liang! This is looking good. A couple of comments:

          1. We also need to fix other locations that are synchronizing on the FSDatasetImpl object e.g. FsVolumeImpl, DirectoryScanner.
          2. Let's move the instrumentation changes to a separate Jira. We can repurpose this just for splitting out the lock. Comments on the instrumentation changes:
            1. We don't need ThreadLocal or threadID-> timestamps map. We are measuring the lock held time so we can save a timestamp just after getting the lock and another timestamp just before releasing the lock. Then diff them with the lock held and log after releasing the lock. We may need to use a thread local approach later if we have a read-write lock in which case there can be multiple concurrent lock holders.
            2. You don't need the if (start == 0 || start2 == 0) checks. These values they can be assumed to be correct now they are initialized in the lock class.
          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the updated patch Chen Liang ! This is looking good. A couple of comments: We also need to fix other locations that are synchronizing on the FSDatasetImpl object e.g. FsVolumeImpl , DirectoryScanner . Let's move the instrumentation changes to a separate Jira. We can repurpose this just for splitting out the lock. Comments on the instrumentation changes: We don't need ThreadLocal or threadID-> timestamps map. We are measuring the lock held time so we can save a timestamp just after getting the lock and another timestamp just before releasing the lock. Then diff them with the lock held and log after releasing the lock. We may need to use a thread local approach later if we have a read-write lock in which case there can be multiple concurrent lock holders. You don't need the if (start == 0 || start2 == 0) checks. These values they can be assumed to be correct now they are initialized in the lock class.
          Hide
          vagarychen Chen Liang added a comment -

          Thank you Arpit Agarwal very much for the review! A few things I would like to clarify though:

          1. by "fix the other locations", did you mean fixing places like such?:
          synchronized(dataset)

          { ... }

          As dataset itself has been refactored with a separate lock, having this synchronized cal meanswe would have two locks here. I have been thinking about situations like this. And I think we need dataset to expose lock acquire and release to those callers. What do you think?

          2. Regarding the ThreadLocal variables, you have a good point. But I was trying to take into account lock acquire time and lock release time there, which requires recording time before acquiring lock and after releasing lock and this is ThreadLocal is all about. Do you have any comments on this? e.g. are these values worth recording?

          3. And you are totally right that I don't need the equals zero checks, thanks for pointing it out!

          Show
          vagarychen Chen Liang added a comment - Thank you Arpit Agarwal very much for the review! A few things I would like to clarify though: 1. by "fix the other locations", did you mean fixing places like such?: synchronized(dataset) { ... } As dataset itself has been refactored with a separate lock, having this synchronized cal meanswe would have two locks here. I have been thinking about situations like this. And I think we need dataset to expose lock acquire and release to those callers. What do you think? 2. Regarding the ThreadLocal variables, you have a good point. But I was trying to take into account lock acquire time and lock release time there, which requires recording time before acquiring lock and after releasing lock and this is ThreadLocal is all about. Do you have any comments on this? e.g. are these values worth recording? 3. And you are totally right that I don't need the equals zero checks, thanks for pointing it out!
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          And I think we need dataset to expose lock acquire and release to those callers. What do you think?

          Hi Chen Liang, yes that's correct. We'd need FsDatasetImpl (and perhaps FsDatasetSpi) to expose locking routines.

          I was trying to take into account lock acquire time and lock release time there, which requires recording time before acquiring lock and after releasing lock and this is ThreadLocal is all about.

          Yes I agree with the idea, if we want to incorporate lock-held/release time we'd need either thread locals or the map you talked about. But we can skip measuring those values for simplicity and to avoid questions like thread local overhead. We can consider adding more measurements later.

          Show
          arpitagarwal Arpit Agarwal added a comment - And I think we need dataset to expose lock acquire and release to those callers. What do you think? Hi Chen Liang , yes that's correct. We'd need FsDatasetImpl (and perhaps FsDatasetSpi) to expose locking routines. I was trying to take into account lock acquire time and lock release time there, which requires recording time before acquiring lock and after releasing lock and this is ThreadLocal is all about. Yes I agree with the idea, if we want to incorporate lock-held/release time we'd need either thread locals or the map you talked about. But we can skip measuring those values for simplicity and to avoid questions like thread local overhead. We can consider adding more measurements later.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 26s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 46s the patch passed
          +1 javac 0m 46s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 16 new + 109 unchanged - 11 fixed = 125 total (was 120)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 0m 55s the patch passed
          -1 unit 66m 10s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          86m 45s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.datanode.TestBlockRecovery
          Timed out junit tests org.apache.hadoop.hdfs.TestReadWhileWriting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821459/HDFS-10682.006.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 81bdc8af99e1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9f473cf
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16282/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16282/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16282/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16282/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 26s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 16 new + 109 unchanged - 11 fixed = 125 total (was 120) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 0m 55s the patch passed -1 unit 66m 10s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 86m 45s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.datanode.TestBlockRecovery Timed out junit tests org.apache.hadoop.hdfs.TestReadWhileWriting Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821459/HDFS-10682.006.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 81bdc8af99e1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f473cf Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16282/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16282/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16282/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16282/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fenghua_hu Fenghua Hu added a comment -

          Arpit/Liang,
          Looks like there is one JIRA(https://issues.apache.org/jira/browse/HDFS-9668) to address the big lock issue, maybe we should we relate them?

          Show
          fenghua_hu Fenghua Hu added a comment - Arpit/Liang, Looks like there is one JIRA( https://issues.apache.org/jira/browse/HDFS-9668 ) to address the big lock issue, maybe we should we relate them?
          Hide
          vagarychen Chen Liang added a comment -

          Thanks for the feedbacks, Arpit! Will upload another patch to reflect this.

          Show
          vagarychen Chen Liang added a comment - Thanks for the feedbacks, Arpit! Will upload another patch to reflect this.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Fenghua Hu, thanks for the heads up. I've linked HDFS-9968. That one is going to be a more difficult task. Here we just plan to add instrumentation around the lock so we get a better sense of how bad the locking problem is in real clusters.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Fenghua Hu , thanks for the heads up. I've linked HDFS-9968. That one is going to be a more difficult task. Here we just plan to add instrumentation around the lock so we get a better sense of how bad the locking problem is in real clusters.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 7m 37s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 530 unchanged - 11 fixed = 548 total (was 541)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 54s the patch passed
          +1 unit 56m 54s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          77m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821688/HDFS-10682.007.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1fadd43e7add 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 954465e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16295/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16295/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16295/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 7m 37s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 530 unchanged - 11 fixed = 548 total (was 541) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 54s the patch passed +1 unit 56m 54s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 77m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821688/HDFS-10682.007.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1fadd43e7add 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 954465e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16295/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16295/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16295/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for working through the patch iterations Chen Liang! The v8 patch is looking good. Just a few minor comments:

          1. AutoCloseableLock should not extend ReentrantLock, it already contains a lock object.
          2. Unused function FsDatasetSpi#releaseDatasetLock and descendants should be removed.

          +1 with these addressed.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for working through the patch iterations Chen Liang ! The v8 patch is looking good. Just a few minor comments: AutoCloseableLock should not extend ReentrantLock, it already contains a lock object. Unused function FsDatasetSpi#releaseDatasetLock and descendants should be removed. +1 with these addressed.
          Hide
          vagarychen Chen Liang added a comment -

          Thanks for the review Arpit Agarwal! Uploaded another patch to fix these.

          Show
          vagarychen Chen Liang added a comment - Thanks for the review Arpit Agarwal ! Uploaded another patch to fix these.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 6m 34s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 530 unchanged - 11 fixed = 530 total (was 541)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 51s the patch passed
          -1 unit 60m 51s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          79m 33s



          Reason Tests
          Failed junit tests hadoop.tracing.TestTracing
            hadoop.hdfs.TestFileChecksum



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821885/HDFS-10682.009.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d49d7e1e4d1b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 580a833
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16311/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16311/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16311/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 6m 34s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 530 unchanged - 11 fixed = 530 total (was 541) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 51s the patch passed -1 unit 60m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 79m 33s Reason Tests Failed junit tests hadoop.tracing.TestTracing   hadoop.hdfs.TestFileChecksum Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821885/HDFS-10682.009.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d49d7e1e4d1b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 580a833 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16311/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16311/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16311/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vagarychen Chen Liang added a comment -

          The failed tests are unrelated.

          Show
          vagarychen Chen Liang added a comment - The failed tests are unrelated.
          Hide
          vagarychen Chen Liang added a comment -

          Thanks to Arpit Agarwal for the advise! I moved the separate lock class to hadoop common utils and filed HADOOP-13466 to keep track of it. Will later on update this JIRA to just use the lock class.

          Show
          vagarychen Chen Liang added a comment - Thanks to Arpit Agarwal for the advise! I moved the separate lock class to hadoop common utils and filed HADOOP-13466 to keep track of it. Will later on update this JIRA to just use the lock class.
          Hide
          vagarychen Chen Liang added a comment -

          The lock class is committed as in HADOOP-13466. The updated patch excludes the lock from itself but instead uses the lock class.

          Show
          vagarychen Chen Liang added a comment - The lock class is committed as in HADOOP-13466 . The updated patch excludes the lock from itself but instead uses the lock class.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 6m 55s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 40s the patch passed
          +1 javac 0m 40s the patch passed
          +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 530 unchanged - 11 fixed = 530 total (was 541)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 59m 48s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          79m 0s



          Reason Tests
          Failed junit tests hadoop.tracing.TestTracing
            hadoop.security.TestRefreshUserMappings



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822208/HDFS-10682.010.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 11329eed7e9b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e0d131f
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16332/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16332/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 6m 55s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 40s the patch passed +1 javac 0m 40s the patch passed +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 530 unchanged - 11 fixed = 530 total (was 541) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 59m 48s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 79m 0s Reason Tests Failed junit tests hadoop.tracing.TestTracing   hadoop.security.TestRefreshUserMappings Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822208/HDFS-10682.010.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11329eed7e9b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0d131f Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16332/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16332/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vagarychen Chen Liang added a comment - - edited

          The failed tests are unrelated and seem to be tests that fail frequently recently, see:
          HADOOP-13469 which tracks TestRefreshUserMappings fail
          HDFS-10732 which tracks TestTracing fail

          Show
          vagarychen Chen Liang added a comment - - edited The failed tests are unrelated and seem to be tests that fail frequently recently, see: HADOOP-13469 which tracks TestRefreshUserMappings fail HDFS-10732 which tracks TestTracing fail
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 lgtm. I will commit it shortly.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 lgtm. I will commit it shortly.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I've committed this to trunk. Thanks for the contribution Chen Liang! Also would you consider posting a branch-2 patch as there were a bunch of conflicts? Thanks.

          Show
          arpitagarwal Arpit Agarwal added a comment - I've committed this to trunk. Thanks for the contribution Chen Liang ! Also would you consider posting a branch-2 patch as there were a bunch of conflicts? Thanks.
          Hide
          vagarychen Chen Liang added a comment -

          Thanks Arpit Agarwal! I attached a patch for branch-2.8, but I'm not sure whether will Jenkins checks it as this JIRA is in RESOLVED status.

          Show
          vagarychen Chen Liang added a comment - Thanks Arpit Agarwal ! I attached a patch for branch-2.8, but I'm not sure whether will Jenkins checks it as this JIRA is in RESOLVED status.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Chen Liang, I've reopened the Jira. I think Jenkins cannot check branch-2.8 patches but it can check branch-2 patches. If you rename it to HDFS-10682-branch-2.001.patch then Jenkins will pick it up.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Chen Liang , I've reopened the Jira. I think Jenkins cannot check branch-2.8 patches but it can check branch-2 patches. If you rename it to HDFS-10682 -branch-2.001.patch then Jenkins will pick it up.
          Hide
          vagarychen Chen Liang added a comment -

          Was not aware of this.Thanks Arpit Agarwal.

          Show
          vagarychen Chen Liang added a comment - Was not aware of this.Thanks Arpit Agarwal .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HDFS-10682 does not apply to branch-2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822684/HDFS-10682-branch-2.001.patch
          JIRA Issue HDFS-10682
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16345/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HDFS-10682 does not apply to branch-2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822684/HDFS-10682-branch-2.001.patch JIRA Issue HDFS-10682 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16345/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vagarychen Chen Liang added a comment -

          The patch was originally generated based on branch-2.8, it seems it is fine with branch-2.8 but not with branch-2. Will fix this and put another patch.

          Show
          vagarychen Chen Liang added a comment - The patch was originally generated based on branch-2.8, it seems it is fine with branch-2.8 but not with branch-2. Will fix this and put another patch.
          Hide
          vagarychen Chen Liang added a comment -

          re-submitted a patch for branch-2

          Show
          vagarychen Chen Liang added a comment - re-submitted a patch for branch-2
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 11m 6s branch-2.8 passed
          +1 compile 0m 52s branch-2.8 passed with JDK v1.8.0_101
          +1 compile 0m 48s branch-2.8 passed with JDK v1.7.0_101
          +1 checkstyle 0m 28s branch-2.8 passed
          +1 mvnsite 0m 57s branch-2.8 passed
          +1 mvneclipse 0m 19s branch-2.8 passed
          +1 findbugs 2m 15s branch-2.8 passed
          +1 javadoc 1m 4s branch-2.8 passed with JDK v1.8.0_101
          +1 javadoc 1m 44s branch-2.8 passed with JDK v1.7.0_101
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.8.0_101
          +1 javac 0m 41s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.7.0_101
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 365 unchanged - 14 fixed = 369 total (was 379)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 19s the patch passed
          +1 javadoc 1m 4s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 36s the patch passed with JDK v1.7.0_101
          -1 unit 49m 33s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          136m 35s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestFileAppendRestart
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.hdfs.server.datanode.TestBlockRecovery
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.security.TestRefreshUserMappings
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
            hadoop.hdfs.server.datanode.TestTransferRbw
          JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFsck
          JDK v1.7.0_101 Failed junit tests hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.hdfs.TestWriteRead
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.TestFileAppendRestart
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.hdfs.server.datanode.TestBlockRecovery
            hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.security.TestRefreshUserMappings
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
            hadoop.hdfs.server.datanode.TestTransferRbw
          JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFsck



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5af2af1
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822680/HDFS-10682.branch-2.8.001.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1f3f3b709d44 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.8 / 7b042c5
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16344/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16344/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16344/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16344/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 11m 6s branch-2.8 passed +1 compile 0m 52s branch-2.8 passed with JDK v1.8.0_101 +1 compile 0m 48s branch-2.8 passed with JDK v1.7.0_101 +1 checkstyle 0m 28s branch-2.8 passed +1 mvnsite 0m 57s branch-2.8 passed +1 mvneclipse 0m 19s branch-2.8 passed +1 findbugs 2m 15s branch-2.8 passed +1 javadoc 1m 4s branch-2.8 passed with JDK v1.8.0_101 +1 javadoc 1m 44s branch-2.8 passed with JDK v1.7.0_101 +1 mvninstall 0m 51s the patch passed +1 compile 0m 41s the patch passed with JDK v1.8.0_101 +1 javac 0m 41s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_101 +1 javac 0m 43s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 365 unchanged - 14 fixed = 369 total (was 379) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 36s the patch passed with JDK v1.7.0_101 -1 unit 49m 33s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 136m 35s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.TestEncryptedTransfer   hadoop.security.TestRefreshUserMappings   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.hdfs.server.datanode.TestTransferRbw JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFsck JDK v1.7.0_101 Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.TestWriteRead   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.TestEncryptedTransfer   hadoop.security.TestRefreshUserMappings   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.hdfs.server.datanode.TestTransferRbw JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFsck Subsystem Report/Notes Docker Image:yetus/hadoop:5af2af1 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822680/HDFS-10682.branch-2.8.001.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1f3f3b709d44 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 7b042c5 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16344/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16344/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16344/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16344/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 6m 37s branch-2 passed
          +1 compile 0m 40s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 42s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 33s branch-2 passed
          +1 mvnsite 0m 52s branch-2 passed
          +1 mvneclipse 0m 16s branch-2 passed
          +1 findbugs 1m 51s branch-2 passed
          +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.8.0_101
          +1 javac 0m 36s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.7.0_101
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 509 unchanged - 11 fixed = 510 total (was 520)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 4s the patch passed
          +1 javadoc 0m 51s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 32s the patch passed with JDK v1.7.0_101
          -1 unit 52m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
          -1 asflicense 0m 22s The patch generated 1 ASF License warnings.
          131m 4s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.datanode.TestTransferRbw
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestFileAppendRestart
            hadoop.security.TestRefreshUserMappings
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.TestDFSClientSocketSize
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.server.datanode.TestBlockRecovery
            hadoop.hdfs.TestBlocksScheduledCounter
            hadoop.hdfs.TestAclsEndToEnd
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.net.TestNetworkTopology
            hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
          JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFsck
          JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestBalancerBandwidth
            hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter
            hadoop.hdfs.server.datanode.TestTransferRbw
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestFileAppendRestart
            hadoop.security.TestRefreshUserMappings
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.web.TestHftpFileSystem
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.TestXAttrConfigFlag
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.TestCrcCorruption
            hadoop.tracing.TestTracingShortCircuitLocalRead
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.datanode.TestBlockRecovery
            hadoop.hdfs.TestParallelShortCircuitReadUnCached
            hadoop.hdfs.TestAppendDifferentChecksum
            hadoop.hdfs.TestFileAppend4
            hadoop.fs.TestGlobPaths
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
          JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFsck



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822697/HDFS-10682-branch-2.002.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b57c2844c50a 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / e7c7015
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16347/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16347/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16347/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16347/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 6m 37s branch-2 passed +1 compile 0m 40s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 42s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 33s branch-2 passed +1 mvnsite 0m 52s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 findbugs 1m 51s branch-2 passed +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 44s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_101 +1 javac 0m 36s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_101 +1 javac 0m 41s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 509 unchanged - 11 fixed = 510 total (was 520) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 51s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 32s the patch passed with JDK v1.7.0_101 -1 unit 52m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_101. -1 asflicense 0m 22s The patch generated 1 ASF License warnings. 131m 4s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.datanode.TestTransferRbw   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.tracing.TestTracing   hadoop.hdfs.TestFileAppendRestart   hadoop.security.TestRefreshUserMappings   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.TestDFSClientSocketSize   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.TestBlocksScheduledCounter   hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.TestEncryptedTransfer   hadoop.net.TestNetworkTopology   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFsck JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestBalancerBandwidth   hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter   hadoop.hdfs.server.datanode.TestTransferRbw   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.tracing.TestTracing   hadoop.hdfs.TestFileAppendRestart   hadoop.security.TestRefreshUserMappings   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.web.TestHftpFileSystem   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.TestXAttrConfigFlag   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestCrcCorruption   hadoop.tracing.TestTracingShortCircuitLocalRead   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.hdfs.TestFileAppend4   hadoop.fs.TestGlobPaths   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFsck Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822697/HDFS-10682-branch-2.002.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b57c2844c50a 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e7c7015 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16347/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16347/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16347/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16347/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vagarychen Chen Liang added a comment -

          Somehow there is one call missed in a method... Which breaks all the places calling it. Fixed in this patch.

          Show
          vagarychen Chen Liang added a comment - Somehow there is one call missed in a method... Which breaks all the places calling it. Fixed in this patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 18m 57s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 11m 45s branch-2 passed
          +1 compile 0m 49s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 48s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 35s branch-2 passed
          +1 mvnsite 1m 1s branch-2 passed
          +1 mvneclipse 0m 19s branch-2 passed
          +1 findbugs 2m 4s branch-2 passed
          +1 javadoc 0m 58s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 39s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.8.0_101
          +1 javac 0m 42s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_101
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 508 unchanged - 11 fixed = 509 total (was 519)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 17s the patch passed
          +1 javadoc 0m 58s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 38s the patch passed with JDK v1.7.0_101
          -1 unit 73m 0s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          193m 7s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822840/HDFS-10682-branch-2.003.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 95f033fae278 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 2df34ab
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16370/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16370/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16370/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 18m 57s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 11m 45s branch-2 passed +1 compile 0m 49s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 48s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 35s branch-2 passed +1 mvnsite 1m 1s branch-2 passed +1 mvneclipse 0m 19s branch-2 passed +1 findbugs 2m 4s branch-2 passed +1 javadoc 0m 58s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 39s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_101 +1 javac 0m 42s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_101 +1 javac 0m 42s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 508 unchanged - 11 fixed = 509 total (was 519) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 17s the patch passed +1 javadoc 0m 58s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 38s the patch passed with JDK v1.7.0_101 -1 unit 73m 0s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 193m 7s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822840/HDFS-10682-branch-2.003.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 95f033fae278 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 2df34ab Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16370/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16370/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16370/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16370/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jingcheng.du@intel.com Jingcheng Du added a comment -

          Thanks a lot Fenghua Hu, and thanks Arpit Agarwal for linking HDFS-9668.
          According to Colin's suggestion, the changes would be minor.
          I updated only a few methods in FsDatasetImpl and moved the disk operations out of the synchronized block, and planned to post it soon after merging this JIRA HDFS-10682.

          Show
          jingcheng.du@intel.com Jingcheng Du added a comment - Thanks a lot Fenghua Hu , and thanks Arpit Agarwal for linking HDFS-9668 . According to Colin's suggestion, the changes would be minor. I updated only a few methods in FsDatasetImpl and moved the disk operations out of the synchronized block, and planned to post it soon after merging this JIRA HDFS-10682 .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 7m 10s branch-2 passed
          +1 compile 0m 44s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 44s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 32s branch-2 passed
          +1 mvnsite 0m 58s branch-2 passed
          +1 mvneclipse 0m 16s branch-2 passed
          +1 findbugs 2m 8s branch-2 passed
          +1 javadoc 1m 3s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 41s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 44s the patch passed with JDK v1.8.0_101
          +1 javac 0m 44s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.7.0_101
          +1 javac 0m 46s the patch passed
          +1 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 509 unchanged - 11 fixed = 509 total (was 520)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 18s the patch passed
          +1 javadoc 1m 0s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 39s the patch passed with JDK v1.7.0_101
          +1 unit 47m 46s hadoop-hdfs in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          128m 39s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823105/HDFS-10682-branch-2.004.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 71cb77255a66 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / f7ac0fd
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16388/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16388/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 7m 10s branch-2 passed +1 compile 0m 44s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 44s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 32s branch-2 passed +1 mvnsite 0m 58s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 findbugs 2m 8s branch-2 passed +1 javadoc 1m 3s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 41s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 51s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_101 +1 javac 0m 44s the patch passed +1 compile 0m 46s the patch passed with JDK v1.7.0_101 +1 javac 0m 46s the patch passed +1 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 509 unchanged - 11 fixed = 509 total (was 520) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 18s the patch passed +1 javadoc 1m 0s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 39s the patch passed with JDK v1.7.0_101 +1 unit 47m 46s hadoop-hdfs in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 128m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823105/HDFS-10682-branch-2.004.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 71cb77255a66 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / f7ac0fd Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16388/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16388/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Chen Liang, thanks for the branch-2 patch. Couple of comments on the branch-2 v4 patch:

          1. Missing changes to FsVolumeImpl.java which still synchronizes on the dataset object.
          2. BlockSender also needs to be fixed. It looks like we missed this occurrence in trunk so I'll file a separate Jira to fix it for trunk.
                  synchronized(datanode.data) { 
                    replica = getReplica(block, datanode);
                    replicaVisibleLength = replica.getVisibleLength();
                  }
            
          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Chen Liang , thanks for the branch-2 patch. Couple of comments on the branch-2 v4 patch: Missing changes to FsVolumeImpl.java which still synchronizes on the dataset object. BlockSender also needs to be fixed. It looks like we missed this occurrence in trunk so I'll file a separate Jira to fix it for trunk. synchronized (datanode.data) { replica = getReplica(block, datanode); replicaVisibleLength = replica.getVisibleLength(); }
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Filed HDFS-10773.

          Show
          arpitagarwal Arpit Agarwal added a comment - Filed HDFS-10773 .
          Hide
          vagarychen Chen Liang added a comment -

          Thanks Arpit Agarwal for pointing out! Will fix these.

          Show
          vagarychen Chen Liang added a comment - Thanks Arpit Agarwal for pointing out! Will fix these.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the updated patch. Can you please include the BlockSender fix in the branch-2 patch here itself?

          HDFS-10773 can be limited to just trunk.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the updated patch. Can you please include the BlockSender fix in the branch-2 patch here itself? HDFS-10773 can be limited to just trunk.
          Hide
          vagarychen Chen Liang added a comment -

          Good point! Just uploaded an updated patch.

          Show
          vagarychen Chen Liang added a comment - Good point! Just uploaded an updated patch.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the branch-2 v6 patch, pending Jenkins.

          Thanks Chen Liang.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the branch-2 v6 patch, pending Jenkins. Thanks Chen Liang .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 6m 54s branch-2 passed
          +1 compile 0m 42s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 43s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 0m 35s branch-2 passed
          +1 mvnsite 0m 53s branch-2 passed
          +1 mvneclipse 0m 16s branch-2 passed
          +1 findbugs 1m 57s branch-2 passed
          +1 javadoc 0m 56s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 38s branch-2 passed with JDK v1.7.0_111
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_101
          +1 javac 0m 37s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_111
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 561 unchanged - 11 fixed = 561 total (was 572)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 11s the patch passed
          +1 javadoc 0m 53s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 31s the patch passed with JDK v1.7.0_111
          +1 unit 48m 12s hadoop-hdfs in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          138m 38s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824227/HDFS-10682-branch-2.006.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 44ba74d06440 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 2486c4c
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16457/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16457/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 13m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 6m 54s branch-2 passed +1 compile 0m 42s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 43s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 35s branch-2 passed +1 mvnsite 0m 53s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 findbugs 1m 57s branch-2 passed +1 javadoc 0m 56s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 38s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 46s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_101 +1 javac 0m 37s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_111 +1 javac 0m 42s the patch passed +1 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 561 unchanged - 11 fixed = 561 total (was 572) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 11s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 31s the patch passed with JDK v1.7.0_111 +1 unit 48m 12s hadoop-hdfs in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 138m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824227/HDFS-10682-branch-2.006.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 44ba74d06440 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 2486c4c Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16457/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16457/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Committed to branch-2 and branch-2.8. Thank you for the contribution Chen Liang.

          Show
          arpitagarwal Arpit Agarwal added a comment - Committed to branch-2 and branch-2.8. Thank you for the contribution Chen Liang .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 19m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 7m 21s branch-2 passed
          +1 compile 0m 46s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 49s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 0m 43s branch-2 passed
          +1 mvnsite 1m 4s branch-2 passed
          +1 mvneclipse 0m 17s branch-2 passed
          +1 findbugs 2m 11s branch-2 passed
          +1 javadoc 1m 2s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 39s branch-2 passed with JDK v1.7.0_111
          +1 mvninstall 0m 55s the patch passed
          +1 compile 0m 45s the patch passed with JDK v1.8.0_101
          +1 javac 0m 45s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.7.0_111
          +1 javac 0m 46s the patch passed
          +1 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 561 unchanged - 11 fixed = 561 total (was 572)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 13s the patch passed
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 40s the patch passed with JDK v1.7.0_111
          -1 unit 70m 13s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          188m 33s



          Reason Tests
          JDK v1.7.0_111 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824227/HDFS-10682-branch-2.006.patch
          JIRA Issue HDFS-10682
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f324c8895fba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 2486c4c
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16455/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16455/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16455/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 19m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 7m 21s branch-2 passed +1 compile 0m 46s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 49s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 43s branch-2 passed +1 mvnsite 1m 4s branch-2 passed +1 mvneclipse 0m 17s branch-2 passed +1 findbugs 2m 11s branch-2 passed +1 javadoc 1m 2s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 39s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 55s the patch passed +1 compile 0m 45s the patch passed with JDK v1.8.0_101 +1 javac 0m 45s the patch passed +1 compile 0m 46s the patch passed with JDK v1.7.0_111 +1 javac 0m 46s the patch passed +1 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 561 unchanged - 11 fixed = 561 total (was 572) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 13s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 40s the patch passed with JDK v1.7.0_111 -1 unit 70m 13s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 188m 33s Reason Tests JDK v1.7.0_111 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824227/HDFS-10682-branch-2.006.patch JIRA Issue HDFS-10682 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f324c8895fba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 2486c4c Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16455/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16455/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16455/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vagarychen Chen Liang added a comment -

          Thanks Arpit Agarwal!

          Somehow Jenkins ran on the patch v6 a second time and ran into a timed-out test this time. This is unrelated and I was not able to reproduce it either.

          Show
          vagarychen Chen Liang added a comment - Thanks Arpit Agarwal ! Somehow Jenkins ran on the patch v6 a second time and ran into a timed-out test this time. This is unrelated and I was not able to reproduce it either.
          Hide
          fenghua_hu Fenghua Hu added a comment - - edited

          In FsDatasetImpl#FsDatasetImpl() and FsDatasetImpl#addVolume():
          volumeMap = new ReplicaMap(this);
          and
          ReplicaMap tempVolumeMap = new ReplicaMap(this);

          "this" is used as synchronization object:

          ReplicaMap(Object mutex) {
          if (mutex == null)

          { throw new HadoopIllegalArgumentException( "Object to synchronize on cannot be null"); }

          this.mutex = mutex;
          }

          ReplicaMap uses synchronized(mutex)

          {...}

          for synchronization. Do we need change it accordingly?
          Chen Liang Arpit Agarwal

          Show
          fenghua_hu Fenghua Hu added a comment - - edited In FsDatasetImpl#FsDatasetImpl() and FsDatasetImpl#addVolume(): volumeMap = new ReplicaMap(this); and ReplicaMap tempVolumeMap = new ReplicaMap(this); "this" is used as synchronization object: ReplicaMap(Object mutex) { if (mutex == null) { throw new HadoopIllegalArgumentException( "Object to synchronize on cannot be null"); } this.mutex = mutex; } ReplicaMap uses synchronized(mutex) {...} for synchronization. Do we need change it accordingly? Chen Liang Arpit Agarwal
          Hide
          vagarychen Chen Liang added a comment -

          Hi Fenghua Hu, that is a good catch, thanks! I think we do need to change these places just as you said.

          Show
          vagarychen Chen Liang added a comment - Hi Fenghua Hu , that is a good catch, thanks! I think we do need to change these places just as you said.
          Hide
          fenghua_hu Fenghua Hu added a comment -

          Chen Liang Actually, I am not sure why ReplicaMap takes a so big object FsDatasetImpl as synchronization lock, I think we could use smaller object, thus I have created a new JIRA to change it. Could you please review and comment?
          https://issues.apache.org/jira/browse/HDFS-10804

          Thanks.

          Show
          fenghua_hu Fenghua Hu added a comment - Chen Liang Actually, I am not sure why ReplicaMap takes a so big object FsDatasetImpl as synchronization lock, I think we could use smaller object, thus I have created a new JIRA to change it. Could you please review and comment? https://issues.apache.org/jira/browse/HDFS-10804 Thanks.
          Hide
          vagarychen Chen Liang added a comment -

          I am also thinking of a better way of doing this. Will definitely look at you JIRA, thanks!

          Show
          vagarychen Chen Liang added a comment - I am also thinking of a better way of doing this. Will definitely look at you JIRA, thanks!
          Hide
          xiaochen Xiao Chen added a comment -

          Hi Arpit Agarwal, I couldn't find this in branch-2, only in branch-2.8.
          Could you point me to the branch-2 commit hash? Thanks.

          Show
          xiaochen Xiao Chen added a comment - Hi Arpit Agarwal , I couldn't find this in branch-2, only in branch-2.8. Could you point me to the branch-2 commit hash? Thanks.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the catch Xiao Chen, I've cherry-picked from branch-2.8 to branch-2.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the catch Xiao Chen , I've cherry-picked from branch-2.8 to branch-2.

            People

            • Assignee:
              vagarychen Chen Liang
              Reporter:
              vagarychen Chen Liang
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development