Details

    • Hadoop Flags:
      Reviewed

      Description

      The NameNode logs a message if the FSNamesystem write lock is held by a thread for over 1 second. These messages can be throttled to at one most one per x minutes to avoid potentially filling up NN logs. We can also log the number of suppressed notices since the last log message.

      1. HDFS-10713.000.patch
        8 kB
        Hanisha Koneru
      2. HDFS-10713.001.patch
        9 kB
        Hanisha Koneru
      3. HDFS-10713.002.patch
        9 kB
        Hanisha Koneru
      4. HDFS-10713.003.patch
        15 kB
        Hanisha Koneru
      5. HDFS-10713.004.patch
        15 kB
        Hanisha Koneru
      6. HDFS-10713.005.patch
        16 kB
        Hanisha Koneru
      7. HDFS-10713.006.patch
        15 kB
        Hanisha Koneru
      8. HDFS-10713.007.patch
        19 kB
        Hanisha Koneru
      9. HDFS-10713.008.patch
        19 kB
        Hanisha Koneru
      10. HDFS-10713.009.patch
        19 kB
        Hanisha Koneru
      11. HDFS-10713.010.patch
        19 kB
        Hanisha Koneru
      12. HDFS-10713.011.patch
        19 kB
        Hanisha Koneru
      13. HDFS-10713.012.patch
        18 kB
        Hanisha Koneru
      14. HDFS-10713.branch-2.000.patch
        18 kB
        Hanisha Koneru

        Issue Links

          Activity

          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the backport Zhe Zhang.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the backport Zhe Zhang .
          Hide
          zhz Zhe Zhang added a comment -

          I backported this change to branch-2.7 since the original read/write lock logging patches are there.

          Show
          zhz Zhe Zhang added a comment - I backported this change to branch-2.7 since the original read/write lock logging patches are there.
          Hide
          xkrogen Erik Krogen added a comment - - edited

          Arpit Agarwal, I'm confident that my HDFS-10817 patch would not have added latency cost since it is just manipulating a ThreadLocal but I agree that there is potential for both this patch and HDFS-10475, which I am working on, to add overheads (and this is part of why the metric logic in HDFS-10475 is disabled by default). I was hoping to do a little bit of microbenchmarking as part of HDFS-10475; I could include this patch in the comparison.

          Show
          xkrogen Erik Krogen added a comment - - edited Arpit Agarwal , I'm confident that my HDFS-10817 patch would not have added latency cost since it is just manipulating a ThreadLocal but I agree that there is potential for both this patch and HDFS-10475 , which I am working on, to add overheads (and this is part of why the metric logic in HDFS-10475 is disabled by default). I was hoping to do a little bit of microbenchmarking as part of HDFS-10475 ; I could include this patch in the comparison.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Since HDFS-10817 and related JIRAs are adding some overhead to a critical section of the NN (literally), has anybody measured throughput/latency differences when compared to branch-2?

          Hi Erik Krogen, do you have any thoughts on this question by Chris?

          Show
          arpitagarwal Arpit Agarwal added a comment - Since HDFS-10817 and related JIRAs are adding some overhead to a critical section of the NN (literally), has anybody measured throughput/latency differences when compared to branch-2? Hi Erik Krogen , do you have any thoughts on this question by Chris?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the branch-2 patch. I've committed this for 2.8.0.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the branch-2 patch. I've committed this for 2.8.0.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 49s branch-2 passed
          +1 compile 0m 46s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 46s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 0m 30s branch-2 passed
          +1 mvnsite 0m 56s branch-2 passed
          +1 mvneclipse 0m 17s branch-2 passed
          +1 findbugs 2m 7s branch-2 passed
          +1 javadoc 1m 3s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 43s branch-2 passed with JDK v1.7.0_111
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.8.0_101
          +1 javac 0m 42s the patch passed
          +1 compile 0m 47s the patch passed with JDK v1.7.0_111
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 203 unchanged - 3 fixed = 204 total (was 206)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 19s the patch passed
          +1 javadoc 0m 57s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_111
          +1 unit 49m 31s hadoop-hdfs in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          146m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830430/HDFS-10713.branch-2.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 31dc7abb6ef4 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 809a45a
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16876/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16876/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16876/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 14m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 49s branch-2 passed +1 compile 0m 46s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 46s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 30s branch-2 passed +1 mvnsite 0m 56s branch-2 passed +1 mvneclipse 0m 17s branch-2 passed +1 findbugs 2m 7s branch-2 passed +1 javadoc 1m 3s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 43s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 49s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_101 +1 javac 0m 42s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_111 +1 javac 0m 47s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 203 unchanged - 3 fixed = 204 total (was 206) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_111 +1 unit 49m 31s hadoop-hdfs in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 146m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830430/HDFS-10713.branch-2.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 31dc7abb6ef4 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 809a45a Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16876/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16876/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16876/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the branch-2 patch, pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the branch-2 patch, pending Jenkins.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10491 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10491/)
          HDFS-10713. Throttle FsNameSystem lock warnings. Contributed by Hanisha (arp: rev 059058f9614613667d5385f76022294e07e140aa)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10491 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10491/ ) HDFS-10713 . Throttle FsNameSystem lock warnings. Contributed by Hanisha (arp: rev 059058f9614613667d5385f76022294e07e140aa) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thank you Arpit Agarwal for reviewing and committing the patch. I have posted a patch for branch-2.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal for reviewing and committing the patch. I have posted a patch for branch-2.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Thanks Hanisha Koneru for the contribution and everyone else for the reviews and discussion. I pushed it to trunk with the following trivial edit to satisfy checkstyle.

          -          && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock, readLockInterval));
          +          && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock,
          +                                                        readLockInterval));
          

          Hanisha if you want to post a branch-2 patch I can commit that too, I think it just needs trivial changes to TestFSNameSystem.java to make variables accessed in the inner classes final since Java 7 won't allow access to non-final vars.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Thanks Hanisha Koneru for the contribution and everyone else for the reviews and discussion. I pushed it to trunk with the following trivial edit to satisfy checkstyle. - && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock, readLockInterval)); + && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock, + readLockInterval)); Hanisha if you want to post a branch-2 patch I can commit that too, I think it just needs trivial changes to TestFSNameSystem.java to make variables accessed in the inner classes final since Java 7 won't allow access to non-final vars.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          The junit tests pass locally.

          • hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            • Intermittent test failure. A bug has already been raised for this: HDFS-10426
          • hadoop.hdfs.server.datanode.TestDataNodeLifeline
            • Intermittent test failure. A bug has already been raised for this: HDFS-9911
          Show
          hanishakoneru Hanisha Koneru added a comment - The junit tests pass locally. hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Intermittent test failure. A bug has already been raised for this: HDFS-10426 hadoop.hdfs.server.datanode.TestDataNodeLifeline Intermittent test failure. A bug has already been raised for this: HDFS-9911
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 33s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 194 unchanged - 3 fixed = 196 total (was 197)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 64m 8s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          84m 34s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.datanode.TestDataNodeLifeline



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829300/HDFS-10713.012.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cf0cccc2940b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 98bdb51
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16801/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16801/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16801/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16801/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 33s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 194 unchanged - 3 fixed = 196 total (was 197) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 64m 8s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 84m 34s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.datanode.TestDataNodeLifeline Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829300/HDFS-10713.012.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cf0cccc2940b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 98bdb51 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16801/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16801/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16801/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16801/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks Chris Douglas for reviewing the patch.

          • I have optimized the do while block as per your suggestion.
          • The setTimeStampOfLastReadLockReport and setTimeStampOfLastWriteLockReport methods are no longer necessary. I have removed them.
          • The Thread.sleep’s at lines 311 and 370 in TestFSNamesystem were removed as they are not required. We are testing for write/read lock held for a short time and hence having 0 lock held time would also work here.
          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks Chris Douglas for reviewing the patch. I have optimized the do while block as per your suggestion. The setTimeStampOfLastReadLockReport and setTimeStampOfLastWriteLockReport methods are no longer necessary. I have removed them. The Thread.sleep’s at lines 311 and 370 in TestFSNamesystem were removed as they are not required. We are testing for write/read lock held for a short time and hence having 0 lock held time would also work here.
          Hide
          chris.douglas Chris Douglas added a comment -

          +0 on v11. Since HDFS-10817 and related JIRAs are adding some overhead to a critical section of the NN (literally), has anybody measured throughput/latency differences when compared to branch-2?

          • Minor:
            +      do {
            +        localLongestReadLock = longestReadLockHeldInterval.get();
            +        if (readLockInterval < localLongestReadLock) {
            +          break;
            +        }
            +      } while (!longestReadLockHeldInterval.compareAndSet(
            +          localLongestReadLock, readLockInterval));
            

            Equivalently

            do {
              localLongestReadLock = longestReadLockHeldInterval.get();
            } while (localLongestReadLock - readLockInterval < 0
                && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock, readLockInterval));
            

            Following guidance on System#nanoTime and avoiding redundant assignment when readLockInterval == localLongestReadLock.

          • Are the @VisibleForTesting methods setTimeStampOfLastReadLockReport and setTimeStampOfLastWriteLockReport necessary for the unit test?
          • Missing replacement call to FakeTimer#advance:
                 fsn.writeLock();
            -    Thread.sleep(writeLockReportingThreshold / 2);
                 fsn.writeUnlock();
            
          Show
          chris.douglas Chris Douglas added a comment - +0 on v11. Since HDFS-10817 and related JIRAs are adding some overhead to a critical section of the NN (literally), has anybody measured throughput/latency differences when compared to branch-2? Minor: + do { + localLongestReadLock = longestReadLockHeldInterval.get(); + if (readLockInterval < localLongestReadLock) { + break; + } + } while (!longestReadLockHeldInterval.compareAndSet( + localLongestReadLock, readLockInterval)); Equivalently do { localLongestReadLock = longestReadLockHeldInterval.get(); } while (localLongestReadLock - readLockInterval < 0 && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock, readLockInterval)); Following guidance on System#nanoTime and avoiding redundant assignment when readLockInterval == localLongestReadLock . Are the @VisibleForTesting methods setTimeStampOfLastReadLockReport and setTimeStampOfLastWriteLockReport necessary for the unit test? Missing replacement call to FakeTimer#advance : fsn.writeLock(); - Thread.sleep(writeLockReportingThreshold / 2); fsn.writeUnlock();
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the v11 patch.

          I'll hold off committing for a couple of days.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the v11 patch. I'll hold off committing for a couple of days.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 24s trunk passed
          +1 compile 0m 43s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 56m 11s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          75m 46s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSShell
            hadoop.hdfs.security.TestDelegationToken



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829073/HDFS-10713.011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 29fb44fc5ac0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f67237c
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16786/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16786/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 24s trunk passed +1 compile 0m 43s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 44s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 56m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 75m 46s Reason Tests Failed junit tests hadoop.hdfs.TestDFSShell   hadoop.hdfs.security.TestDelegationToken Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829073/HDFS-10713.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 29fb44fc5ac0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f67237c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16786/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16786/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks Arpit.
          Removed the duplicated code.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks Arpit. Removed the duplicated code.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          v10 patch lgtm. I think this block was left in by mistake in the second do-while loop in readUnlock, it can be removed.

                  localLongestReadLock = longestReadLockHeldInterval.get();
                  if (readLockInterval > localLongestReadLock) {
                    longestReadLockHeldInterval.compareAndSet(
                        localLongestReadLock, readLockInterval);
                  }
          

          Will hold off committing until next week in case Chris, Erik or others have additional comments.

          Show
          arpitagarwal Arpit Agarwal added a comment - v10 patch lgtm. I think this block was left in by mistake in the second do-while loop in readUnlock, it can be removed. localLongestReadLock = longestReadLockHeldInterval.get(); if (readLockInterval > localLongestReadLock) { longestReadLockHeldInterval.compareAndSet( localLongestReadLock, readLockInterval); } Will hold off committing until next week in case Chris, Erik or others have additional comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 37s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 11s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 1m 50s the patch passed
          +1 javadoc 1m 2s the patch passed
          -1 unit 62m 5s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          81m 15s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828918/HDFS-10713.010.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 709212a55191 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f6f3a44
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16781/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16781/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 9s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 50s the patch passed +1 javadoc 1m 2s the patch passed -1 unit 62m 5s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 81m 15s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828918/HDFS-10713.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 709212a55191 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f6f3a44 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16781/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16781/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 37s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 4s trunk passed
          +1 compile 0m 50s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 59s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 55s the patch passed
          +1 javac 0m 55s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 194 unchanged - 3 fixed = 196 total (was 197)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 54s the patch passed
          +1 unit 78m 23s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          99m 7s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828915/HDFS-10713.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 071d92e9c5a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f6f3a44
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16780/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16780/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16780/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16780/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 37s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 4s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 59s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 194 unchanged - 3 fixed = 196 total (was 197) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 53s the patch passed +1 javadoc 0m 54s the patch passed +1 unit 78m 23s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 99m 7s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828915/HDFS-10713.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 071d92e9c5a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f6f3a44 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16780/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16780/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16780/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16780/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks Arpit. I have addressed the latest comments in v9 patch.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks Arpit. I have addressed the latest comments in v9 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 36s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 49s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 38s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 53s the patch passed
          -1 unit 56m 47s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          75m 18s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSShell



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828876/HDFS-10713.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c0e04a852f20 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b09a03c
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16774/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16774/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 36s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 38s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 9s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 46s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 56m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 75m 18s Reason Tests Failed junit tests hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828876/HDFS-10713.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c0e04a852f20 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b09a03c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16774/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16774/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the updated patch Hanisha. longestReadLockHeldInterval needs its own do-while loop for atomic update (can be before the existing do-while loop).

          Also the three test functions in FsNameSystem (setTimer, setTimeStampOfLastReadLockReport and setTimeStampOfLastWriteLockReport) can be package private.

          LGTM otherwise.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the updated patch Hanisha. longestReadLockHeldInterval needs its own do-while loop for atomic update (can be before the existing do-while loop). Also the three test functions in FsNameSystem (setTimer, setTimeStampOfLastReadLockReport and setTimeStampOfLastWriteLockReport) can be package private. LGTM otherwise.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Fixed Checkstyle and Unit test errors

          Show
          hanishakoneru Hanisha Koneru added a comment - Fixed Checkstyle and Unit test errors
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 46s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 11s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 40s the patch passed
          +1 javac 0m 40s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 194 unchanged - 3 fixed = 201 total (was 197)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 56m 32s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          75m 6s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestSecureNameNode



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828775/HDFS-10713.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e6f2cebfb395 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ec3ea18
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16764/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16764/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16764/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16764/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 40s the patch passed +1 javac 0m 40s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 194 unchanged - 3 fixed = 201 total (was 197) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 56m 32s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 75m 6s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestSecureNameNode Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828775/HDFS-10713.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e6f2cebfb395 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ec3ea18 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16764/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16764/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16764/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16764/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks Erik Krogen, Chris Douglas and Andrew Wang for the review and comments. I have updated the patch with the following changes:
          1. Using Timer.MonoticNow() instead of Time.MonotonicNow() to avoid sleeping in tests.
          2. Replaced synchronized block in readUnlock() with logic from HDFS-10742 to avoid using locks.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks Erik Krogen , Chris Douglas and Andrew Wang for the review and comments. I have updated the patch with the following changes: 1. Using Timer.MonoticNow() instead of Time.MonotonicNow() to avoid sleeping in tests. 2. Replaced synchronized block in readUnlock() with logic from HDFS-10742 to avoid using locks.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Hanisha. I had the same review comment as Erik above about mocking to avoid sleeping in tests. Would prefer if we addressed it here so this patch is standalone.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Hanisha. I had the same review comment as Erik above about mocking to avoid sleeping in tests. Would prefer if we addressed it here so this patch is standalone.
          Hide
          chris.douglas Chris Douglas added a comment -

          Using a synchronized block after releasing the read lock will limit read concurrency. Can this use the pattern from HDFS-10742 that doesn't use locks?

          Show
          chris.douglas Chris Douglas added a comment - Using a synchronized block after releasing the read lock will limit read concurrency. Can this use the pattern from HDFS-10742 that doesn't use locks?
          Hide
          xkrogen Erik Krogen added a comment -

          One comment on this is that it would be ideal to change it to use o.a.h.util.Timer.monotonicNow() instead of Time.monotonicNow() to allow for the unit tests to complete quickly without real sleeps, especially now that we have a longer sleep for the reporting threshold. However I plan to do this as part of the work in progress in HDFS-10475 so I wouldn't worry about it here.

          Good work! Thanks for incorporating my read lock reporting changes into this throttling.

          Show
          xkrogen Erik Krogen added a comment - One comment on this is that it would be ideal to change it to use o.a.h.util.Timer.monotonicNow() instead of Time.monotonicNow() to allow for the unit tests to complete quickly without real sleeps, especially now that we have a longer sleep for the reporting threshold. However I plan to do this as part of the work in progress in HDFS-10475 so I wouldn't worry about it here. Good work! Thanks for incorporating my read lock reporting changes into this throttling.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          The unit test TestDFSUpgradeWithHA fails intermittently. Local rerun is successful. A bug has already been raised for this: HDFS-10644.

          Show
          hanishakoneru Hanisha Koneru added a comment - The unit test TestDFSUpgradeWithHA fails intermittently. Local rerun is successful. A bug has already been raised for this: HDFS-10644 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 4s trunk passed
          +1 compile 0m 53s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 1m 5s trunk passed
          +1 mvninstall 1m 3s the patch passed
          +1 compile 1m 7s the patch passed
          +1 javac 1m 7s the patch passed
          -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197)
          +1 mvnsite 1m 16s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 36s the patch passed
          +1 javadoc 0m 59s the patch passed
          -1 unit 69m 0s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          92m 34s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828309/HDFS-10713.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 514ff61187c2 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 08d8e0b
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16734/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16734/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16734/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16734/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 4s trunk passed +1 compile 0m 53s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 5s trunk passed +1 mvninstall 1m 3s the patch passed +1 compile 1m 7s the patch passed +1 javac 1m 7s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197) +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 36s the patch passed +1 javadoc 0m 59s the patch passed -1 unit 69m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 92m 34s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828309/HDFS-10713.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 514ff61187c2 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 08d8e0b Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16734/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16734/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16734/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16734/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 pending Jenkins.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thank you Arpit Agarwal. I have updated the patch with the fixes.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal . I have updated the patch with the fixes.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Also patch will have to be rebased since HDFS-10742 just introduced dfs.lock.suppress.warning.interval, we can reuse the same setting. Instead of using conf.getLong you will have to use conf.getTimeDuration like.

                    conf.getTimeDuration(
                      DFSConfigKeys.DFS_LOCK_SUPPRESS_WARNING_INTERVAL_KEY,
                      DFSConfigKeys.DFS_LOCK_SUPPRESS_WARNING_INTERVAL_DEFAULT,
                      TimeUnit.MILLISECONDS),
          
          Show
          arpitagarwal Arpit Agarwal added a comment - Also patch will have to be rebased since HDFS-10742 just introduced dfs.lock.suppress.warning.interval , we can reuse the same setting. Instead of using conf.getLong you will have to use conf.getTimeDuration like. conf.getTimeDuration( DFSConfigKeys.DFS_LOCK_SUPPRESS_WARNING_INTERVAL_KEY, DFSConfigKeys.DFS_LOCK_SUPPRESS_WARNING_INTERVAL_DEFAULT, TimeUnit.MILLISECONDS),
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Hanisha Koneru. Few more comments:

          1. Log statement in writeUnlock says Longest read-lock held interval. It should be Longest write-lock held interval.
          2. Move the synchronized block inside the if (needReport && readLockInterval >= this.readLockReportingThreshold) so every lock release does not have to get the readUnlockSyncLock mutex.
          3. Nitpick: readUnlock and writeUnlock both call monotonicNow() twice. We can cache the value returned by the first call to avoid the second call.
          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Hanisha Koneru . Few more comments: Log statement in writeUnlock says Longest read-lock held interval . It should be Longest write-lock held interval . Move the synchronized block inside the if (needReport && readLockInterval >= this.readLockReportingThreshold) so every lock release does not have to get the readUnlockSyncLock mutex. Nitpick: readUnlock and writeUnlock both call monotonicNow() twice. We can cache the value returned by the first call to avoid the second call.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 9s trunk passed
          +1 compile 0m 54s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 56s trunk passed
          +1 javadoc 0m 58s trunk passed
          +1 mvninstall 0m 58s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          -0 checkstyle 0m 38s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 590 unchanged - 1 fixed = 591 total (was 591)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 1s the patch passed
          +1 javadoc 1m 4s the patch passed
          -1 unit 88m 52s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          113m 6s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827636/HDFS-10713.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 8fbdeed16d65 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cae33118
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16676/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16676/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16676/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16676/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 9s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 58s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed -0 checkstyle 0m 38s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 590 unchanged - 1 fixed = 591 total (was 591) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 1s the patch passed +1 javadoc 1m 4s the patch passed -1 unit 88m 52s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 113m 6s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827636/HDFS-10713.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8fbdeed16d65 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cae33118 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16676/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16676/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16676/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16676/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 50s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 591 unchanged - 1 fixed = 592 total (was 592)
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
          +1 javadoc 0m 51s the patch passed
          -1 unit 58m 26s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          77m 34s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Synchronization on Integer in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.readUnlock() At FSNamesystem.java: At FSNamesystem.java:[line 1569]
            Synchronization on FSNamesystem.numReadLockWarningsSuppressed in futile attempt to guard it At FSNamesystem.java:attempt to guard it At FSNamesystem.java:[line 1578]
            Synchronization on FSNamesystem.numReadLockWarningsSuppressed in futile attempt to guard it At FSNamesystem.java:attempt to guard it At FSNamesystem.java:[line 1583]
          Failed junit tests hadoop.hdfs.TestFileChecksum
            hadoop.hdfs.TestCrcCorruption



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827621/HDFS-10713.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 1f3bae201d94 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 20a20c2
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16672/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16672/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16672/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16672/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16672/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 50s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 591 unchanged - 1 fixed = 592 total (was 592) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 javadoc 0m 51s the patch passed -1 unit 58m 26s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 77m 34s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Synchronization on Integer in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.readUnlock() At FSNamesystem.java: At FSNamesystem.java: [line 1569]   Synchronization on FSNamesystem.numReadLockWarningsSuppressed in futile attempt to guard it At FSNamesystem.java:attempt to guard it At FSNamesystem.java: [line 1578]   Synchronization on FSNamesystem.numReadLockWarningsSuppressed in futile attempt to guard it At FSNamesystem.java:attempt to guard it At FSNamesystem.java: [line 1583] Failed junit tests hadoop.hdfs.TestFileChecksum   hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827621/HDFS-10713.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1f3bae201d94 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 20a20c2 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16672/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16672/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/16672/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16672/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16672/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks Arpit Agarwal. I have synchronized the readUnlock warnings.

          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks Arpit Agarwal . I have synchronized the readUnlock warnings.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Thanks for the updated patch Hanisha Koneru. The read lock is not exclusive so the following sequence of operations in readUnlock won't see a consistent state. This block of code probably needs to synchronize on a separate mutex after releasing the read lock.

              if (needReport && readLockInterval >= this.readLockReportingThreshold) {
                if (readLockInterval > longestReadLockHeldInterval) {
                  longestReadLockHeldInterval = readLockInterval;
                }
                if (currentTime - timeStampOfLastReadLockReport > this
                    .lockSuppressWarningInterval) {
                  logReport = true;
                  numSuppressedWarnings = numReadLockWarningsSuppressed;
                  numReadLockWarningsSuppressed = 0;
                  longestLockHeldInterval = longestReadLockHeldInterval;
                  longestReadLockHeldInterval = 0;
                  timeStampOfLastReadLockReport = currentTime;
                } else {
                  numReadLockWarningsSuppressed++;
                }
              }
          
              this.fsLock.readLock().unlock();
          

          I also had some questions about the needReport logic I posted on HDFS-10817.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Thanks for the updated patch Hanisha Koneru . The read lock is not exclusive so the following sequence of operations in readUnlock won't see a consistent state. This block of code probably needs to synchronize on a separate mutex after releasing the read lock. if (needReport && readLockInterval >= this .readLockReportingThreshold) { if (readLockInterval > longestReadLockHeldInterval) { longestReadLockHeldInterval = readLockInterval; } if (currentTime - timeStampOfLastReadLockReport > this .lockSuppressWarningInterval) { logReport = true ; numSuppressedWarnings = numReadLockWarningsSuppressed; numReadLockWarningsSuppressed = 0; longestLockHeldInterval = longestReadLockHeldInterval; longestReadLockHeldInterval = 0; timeStampOfLastReadLockReport = currentTime; } else { numReadLockWarningsSuppressed++; } } this .fsLock.readLock().unlock(); I also had some questions about the needReport logic I posted on HDFS-10817 .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 17s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 48s trunk passed
          +1 javadoc 1m 1s trunk passed
          +1 mvninstall 0m 53s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 590 unchanged - 1 fixed = 591 total (was 591)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 1m 0s the patch passed
          +1 unit 77m 36s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          99m 26s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826950/HDFS-10713.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 9db6c4e88dde 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 07650bc
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16627/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16627/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16627/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 17s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 590 unchanged - 1 fixed = 591 total (was 591) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 49s the patch passed +1 javadoc 1m 0s the patch passed +1 unit 77m 36s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 99m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826950/HDFS-10713.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 9db6c4e88dde 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 07650bc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16627/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16627/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16627/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Capturing and logging the longest lock held interval in each report.
          Added throttling for read lock warnings also.

          Show
          hanishakoneru Hanisha Koneru added a comment - Capturing and logging the longest lock held interval in each report. Added throttling for read lock warnings also.
          Hide
          xkrogen Erik Krogen added a comment -

          In HDFS-10475 we hope to add more detailed metrics for which calls are causing long held locks. We are actively working on this so it should hopefully subsume any useful information that would be gathered by keeping all of the stack traces.

          Show
          xkrogen Erik Krogen added a comment - In HDFS-10475 we hope to add more detailed metrics for which calls are causing long held locks. We are actively working on this so it should hopefully subsume any useful information that would be gathered by keeping all of the stack traces.
          Hide
          liuml07 Mingliang Liu added a comment -

          If we just capture the longest lock held interval but lose the stack trace could that be an acceptable solution. Mingliang Liu, what do you think?

          That makes sense in order to avoid overhead of too frequent capturing the stack trace. If we find in the future we need the longest lock holding stack trace heavily, we can add it back. Before that, operators can reduce the config dfs.lock.suppress.warning.interval.ms for finer-granularity information. Thanks.

          Show
          liuml07 Mingliang Liu added a comment - If we just capture the longest lock held interval but lose the stack trace could that be an acceptable solution. Mingliang Liu, what do you think? That makes sense in order to avoid overhead of too frequent capturing the stack trace. If we find in the future we need the longest lock holding stack trace heavily, we can add it back. Before that, operators can reduce the config dfs.lock.suppress.warning.interval.ms for finer-granularity information. Thanks.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Re capturing the thread stack every time we see a higher lock held interval - worst case we may need to capture ~120 stack traces in 2 minutes. That will probably not add too much overhead but I am not sure.

          If we just capture the longest lock held interval but lose the stack trace could that be an acceptable solution. Mingliang Liu, what do you think?

          Show
          arpitagarwal Arpit Agarwal added a comment - Re capturing the thread stack every time we see a higher lock held interval - worst case we may need to capture ~120 stack traces in 2 minutes. That will probably not add too much overhead but I am not sure. If we just capture the longest lock held interval but lose the stack trace could that be an acceptable solution. Mingliang Liu , what do you think?
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          We can cache the thread's stackTrace and lock time and once a new thread lock warning is encountered, we can log the report for the previous thread. This way all thread's information will be preserved.

          Show
          hanishakoneru Hanisha Koneru added a comment - We can cache the thread's stackTrace and lock time and once a new thread lock warning is encountered, we can log the report for the previous thread. This way all thread's information will be preserved.
          Hide
          jingzhao Jing Zhao added a comment - - edited

          we have a look at the feasibility to expose this information to nntop metrics.

          No... not exposing this information to nntop, but to publish op -> lock-holding-time mapping information to metrics.

          Show
          jingzhao Jing Zhao added a comment - - edited we have a look at the feasibility to expose this information to nntop metrics. No... not exposing this information to nntop, but to publish op -> lock-holding-time mapping information to metrics.
          Hide
          liuml07 Mingliang Liu added a comment - - edited

          One concern is that do we need to dump the longest lock interval information during the suppressed interval, including lock-holding interval and its thread stack. This should reveal more useful information. One extreme example is a case where two threads (t1 and t2) holding the write lock in a sequence: t1-1s, t2-100s, t1-1s, in the current implementation the t2 information will be missing though it's more interesting.

          DFSConfigKeys.java
          414	  // Threshold for how long the write lock warnings must be suppressed
          415	  public static final String DFS_LOCK_SUPPRESS_WARNING_INTERVAL_MS_KEY =
          416	      "dfs.lock.suppress.warning.interval.ms";
          417	  public static final long DFS_LOCK_SUPPRESS_WARNING_INTERVAL_MS_DEFAULT =
          418	      120000L;
          

          And

          hdfs-default.xml
          	2626	<property>
          2627	  <name>dfs.lock.suppress.warning.interval.ms</name>
          2628	  <value>1000</value>
          2629	  <description>The interval between reporting lock warnings.
          2630	  </description>
          2631	</property>
          2632	
          

          I believe the default value of config key dfs.lock.suppress.warning.interval.ms is 2 mins not 1 second?

          Minor comments:

          1. In line 1571, message "Number of suppressed write-lock reports: " + numSuppressedWarnings); should have a "\n" or "\t" before it.
          2. Let's make private final long writeLockReportingThreshold; and private final long writeLockSuppressWarningInterval; final.

          As the following work, Jing Zhao also suggest we have a look at the feasibility to expose this information to nntop metrics.

          Show
          liuml07 Mingliang Liu added a comment - - edited One concern is that do we need to dump the longest lock interval information during the suppressed interval, including lock-holding interval and its thread stack. This should reveal more useful information. One extreme example is a case where two threads (t1 and t2) holding the write lock in a sequence: t1-1s, t2-100s, t1-1s , in the current implementation the t2 information will be missing though it's more interesting. DFSConfigKeys.java 414 // Threshold for how long the write lock warnings must be suppressed 415 public static final String DFS_LOCK_SUPPRESS_WARNING_INTERVAL_MS_KEY = 416 "dfs.lock.suppress.warning.interval.ms" ; 417 public static final long DFS_LOCK_SUPPRESS_WARNING_INTERVAL_MS_DEFAULT = 418 120000L; And hdfs-default.xml 2626 <property> 2627 <name>dfs.lock.suppress.warning.interval.ms</name> 2628 <value>1000</value> 2629 <description>The interval between reporting lock warnings. 2630 </description> 2631 </property> 2632 I believe the default value of config key dfs.lock.suppress.warning.interval.ms is 2 mins not 1 second? Minor comments: In line 1571, message "Number of suppressed write-lock reports: " + numSuppressedWarnings); should have a "\n" or "\t" before it. Let's make private final long writeLockReportingThreshold; and private final long writeLockSuppressWarningInterval; final. As the following work, Jing Zhao also suggest we have a look at the feasibility to expose this information to nntop metrics.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thanks Arpit Agarwal for the review. I have made the above mentioned changes in the new patch (v002)

          Show
          hanishakoneru Hanisha Koneru added a comment - Thanks Arpit Agarwal for the review. I have made the above mentioned changes in the new patch (v002)
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Also we can rename the config key to dfs.lock.suppress.warning.interval.ms. Then it can also be used by the DataNode for HDFS-10742.

          Show
          arpitagarwal Arpit Agarwal added a comment - Also we can rename the config key to dfs.lock.suppress.warning.interval.ms . Then it can also be used by the DataNode for HDFS-10742 .
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Hanisha Koneru. The patch lgtm. The test case needs fixing.

          In TestFsNameSystem.java:

          1. Line 319 - the assertFalse should be assertTrue. Also the comment on line 313 should be fixed.
          2. Line 308 - the sleep can be removed.

          We can also change the writeLockReportingThreshold to 1 millisecond to cut down the test run time.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Hanisha Koneru . The patch lgtm. The test case needs fixing. In TestFsNameSystem.java: Line 319 - the assertFalse should be assertTrue. Also the comment on line 313 should be fixed. Line 308 - the sleep can be removed. We can also change the writeLockReportingThreshold to 1 millisecond to cut down the test run time.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 58s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 57s trunk passed
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 586 unchanged - 1 fixed = 588 total (was 587)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 1m 2s the patch passed
          -1 unit 58m 58s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          80m 40s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.TestDFSZKFailoverController
            hadoop.hdfs.server.namenode.TestFSNamesystem



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826297/HDFS-10713.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 1aab5711ce1f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 20ae1fa
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16586/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16586/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16586/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16586/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 58s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 586 unchanged - 1 fixed = 588 total (was 587) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 2s the patch passed -1 unit 58m 58s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 80m 40s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.server.namenode.TestFSNamesystem Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826297/HDFS-10713.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1aab5711ce1f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 20ae1fa Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16586/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16586/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16586/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16586/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          Thank you Arpit Agarwal for reviewing the patch.
          I have fixed the errors in the new patch (v001).

          Show
          hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal for reviewing the patch. I have fixed the errors in the new patch (v001).
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the patch Hanisha Koneru.

          1. We can do the timestamp checks just before the this.fsLock.writeLock().unlock() call, so the new state is protected by the write lock. The actual logging can be done outside the write lock though. Then you don't need to make the members volatile anymore.
          2. We can cache the value returned by monotonicNow() instead of calling it twice.
          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the patch Hanisha Koneru . We can do the timestamp checks just before the this.fsLock.writeLock().unlock() call, so the new state is protected by the write lock. The actual logging can be done outside the write lock though. Then you don't need to make the members volatile anymore. We can cache the value returned by monotonicNow() instead of calling it twice.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 6s trunk passed
          +1 compile 0m 56s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 57s trunk passed
          +1 javadoc 1m 0s trunk passed
          +1 mvninstall 0m 53s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 586 unchanged - 1 fixed = 595 total (was 587)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 2m 6s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 56s the patch passed
          -1 unit 61m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          83m 35s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Increment of volatile field org.apache.hadoop.hdfs.server.namenode.FSNamesystem.numWriteLockWarningsSuppressed in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock() At FSNamesystem.java:in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock() At FSNamesystem.java:[line 1560]
          Failed junit tests hadoop.tools.TestHdfsConfigFields
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826090/HDFS-10713.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 60b3080209c6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8b57be1
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16570/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16570/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16570/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16570/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16570/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 6s trunk passed +1 compile 0m 56s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 1m 0s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 586 unchanged - 1 fixed = 595 total (was 587) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 2m 6s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 56s the patch passed -1 unit 61m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 83m 35s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Increment of volatile field org.apache.hadoop.hdfs.server.namenode.FSNamesystem.numWriteLockWarningsSuppressed in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock() At FSNamesystem.java:in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock() At FSNamesystem.java: [line 1560] Failed junit tests hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826090/HDFS-10713.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 60b3080209c6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b57be1 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16570/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16570/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/16570/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16570/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16570/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment - - edited

          Patch 000 suppresses the write lock warnings for 2 mins (by default, configurable) and reports the first lock warning after this interval. It also reports the number of suppressed warnings.

          Show
          hanishakoneru Hanisha Koneru added a comment - - edited Patch 000 suppresses the write lock warnings for 2 mins (by default, configurable) and reports the first lock warning after this interval. It also reports the number of suppressed warnings.

            People

            • Assignee:
              hanishakoneru Hanisha Koneru
              Reporter:
              arpitagarwal Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development