Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Add metrics for FSNamesystemLock operations to see, overall, how long each operation is holding the lock for. Use MutableRate metrics for now.

      1. FSLockPerf.java
        9 kB
        Erik Krogen
      2. HDFS-10872.000.patch
        54 kB
        Erik Krogen
      3. HDFS-10872.001.patch
        90 kB
        Erik Krogen
      4. HDFS-10872.002.patch
        90 kB
        Erik Krogen
      5. HDFS-10872.003.patch
        90 kB
        Erik Krogen
      6. HDFS-10872.004.patch
        90 kB
        Erik Krogen
      7. HDFS-10872.005.patch
        85 kB
        Erik Krogen
      8. HDFS-10872.006.patch
        85 kB
        Erik Krogen
      9. HDFS-10872.007.patch
        84 kB
        Erik Krogen
      10. HDFS-10872.008.patch
        222 kB
        Erik Krogen
      11. HDFS-10872.009.patch
        84 kB
        Erik Krogen
      12. HDFS-10872.010.patch
        86 kB
        Erik Krogen
      13. HDFS-10872.011.patch
        86 kB
        Zhe Zhang
      14. HDFS-10872.012.patch
        85 kB
        Erik Krogen
      15. HDFS-10872-branch-2.012.patch
        80 kB
        Erik Krogen
      16. HDFS-10872-branch-2.013.patch
        80 kB
        Erik Krogen
      17. HDFS-10872-branch-2.7.013.patch
        78 kB
        Erik Krogen
      18. jmx-output
        3 kB
        Erik Krogen

        Issue Links

          Activity

          Hide
          xkrogen Erik Krogen added a comment -

          Attaching v000 patch.

          This makes FSNamesystemLock into a MetricsSource. When the lock is released, the amount of time for which the lock was held is recorded as a metric corresponding to the specific operation which caused the lock to be held. This was achieved by manually specifying the operation in calls to FSNamesystemLock.read/writeLock since there is no easy way to tell what operation caused the lock's acquisition from within FSNamesystemLock. Lock releases which do not specify an operation are filed under "OTHER". There are a number of lock usages within BlockManager which have this classification because BlockManager only views the Namesystem API, not FSNamesystem, and I did not want to modify that API. I am open to suggestions here.

          Since this occurs along the critical path and may add overhead, it is disabled by default. To minimize overhead, each thread stores a local copy of the metrics. Every so often, set by a configurable interval, each thread will insert the metrics it has accumulated into a shared MetricsRegistry. I am doing some microbenchmark performance testing now and will post results when they are available.

          Show
          xkrogen Erik Krogen added a comment - Attaching v000 patch. This makes FSNamesystemLock into a MetricsSource . When the lock is released, the amount of time for which the lock was held is recorded as a metric corresponding to the specific operation which caused the lock to be held. This was achieved by manually specifying the operation in calls to FSNamesystemLock.read/writeLock since there is no easy way to tell what operation caused the lock's acquisition from within FSNamesystemLock . Lock releases which do not specify an operation are filed under "OTHER". There are a number of lock usages within BlockManager which have this classification because BlockManager only views the Namesystem API, not FSNamesystem , and I did not want to modify that API. I am open to suggestions here. Since this occurs along the critical path and may add overhead, it is disabled by default. To minimize overhead, each thread stores a local copy of the metrics. Every so often, set by a configurable interval, each thread will insert the metrics it has accumulated into a shared MetricsRegistry . I am doing some microbenchmark performance testing now and will post results when they are available.
          Hide
          xkrogen Erik Krogen added a comment -

          Some comments on performance considerations...

          I am attaching code (FSLockPerf.java) that I used to do somewhat rudimentary microbenchmarking. It's not perfect but hopefully it gives a bit of an idea of what kind of overhead this may incur. If anyone is interested in seeing other numbers let me know and I will do my best to generate them. Note that this feature is disabled by default, so no overhead is incurred for those not actively opting in to the feature.

          I include two different tests, "overall" and "aggTime". In both I focus on the worst case scenario in which all threads are reader threads, i.e. they are not hindered by the Namesystem lock and contend solely on metrics. In both cases I use 200 threads to model what would occur in a highly contested system. Also, all aggregations involve 50 operations, emulating 50 distinct operation types occurring at each thread since the last aggregation, which seems a conservatively high upper bound since most operations are uncommon.

          overall tries to be more wholistic but involves a higher degree of variability since there are actually locks being held and such. This test sets the aggregation interval at various intervals (including completely disabled and a high enough interval that aggregation is never triggered) and tests the overall time it takes each of the 200 threads to complete 500,000 cycles of read lock/unlock (including all metrics-related operations). Over 1,000 iterations I got:

          Agg Interval    Total Time MS (Avg)     Total Time MS (StdDev)
          0       30518   1777
          9999999 30825   1673
          20000   30183   1709
          10000   30272   1681
          5000    30278   1740
          1000    30307   1702
          10      30350   1692
          

          Clearly the metrics processing fits within the noise of locking and such, especially given that the average of the runs with the logic disabled ended up being higher than with the logic enabled. Still, these results were not very satisfying, so I tried to be more specific with aggTime.

          aggTime is the more narrow of the two. I assume the local tracking of metrics is very cheap, simply incrementing a counter within a ThreadLocal, so I focus on the time to do the more expensive aggregate (involving a synchronized method to update the MutableRate metric). First I run a test with only a single thread updating metrics, then do the full 200 threads under a few different conditions: turning on and off aggregation (to get a baseline figure of performance with many threads running), and including a 1-millisecond sleep between operations (to emulate slightly less pessimistic conditions of lock contention). Each thread does 10,000 aggregations and I measure the time per operation; over 10 trials I got:

          10000 aggregation per thread over 100 trials
          Test    Average Time (ns)       Std Dev (ns)
          Single Thread   3107    606
          No Agg, No Wait 551     551
          Agg, No Wait    235850  24059
          No Agg, Wait    1065525 625
          Agg, Wait       1158477 8743
          

          So it seems that even under highly contested conditions an aggregation adds ~100-200 microseconds to the execution path, and without contention only ~3-4 microseconds. Given that a typical aggregation period would be the same as the metrics collection interval, say 10-60 seconds, this seems reasonable for a disabled-by-default feature.

          Show
          xkrogen Erik Krogen added a comment - Some comments on performance considerations... I am attaching code ( FSLockPerf.java ) that I used to do somewhat rudimentary microbenchmarking. It's not perfect but hopefully it gives a bit of an idea of what kind of overhead this may incur. If anyone is interested in seeing other numbers let me know and I will do my best to generate them. Note that this feature is disabled by default, so no overhead is incurred for those not actively opting in to the feature. I include two different tests, "overall" and "aggTime". In both I focus on the worst case scenario in which all threads are reader threads, i.e. they are not hindered by the Namesystem lock and contend solely on metrics. In both cases I use 200 threads to model what would occur in a highly contested system. Also, all aggregations involve 50 operations, emulating 50 distinct operation types occurring at each thread since the last aggregation, which seems a conservatively high upper bound since most operations are uncommon. overall tries to be more wholistic but involves a higher degree of variability since there are actually locks being held and such. This test sets the aggregation interval at various intervals (including completely disabled and a high enough interval that aggregation is never triggered) and tests the overall time it takes each of the 200 threads to complete 500,000 cycles of read lock/unlock (including all metrics-related operations). Over 1,000 iterations I got: Agg Interval Total Time MS (Avg) Total Time MS (StdDev) 0 30518 1777 9999999 30825 1673 20000 30183 1709 10000 30272 1681 5000 30278 1740 1000 30307 1702 10 30350 1692 Clearly the metrics processing fits within the noise of locking and such, especially given that the average of the runs with the logic disabled ended up being higher than with the logic enabled. Still, these results were not very satisfying, so I tried to be more specific with aggTime. aggTime is the more narrow of the two. I assume the local tracking of metrics is very cheap, simply incrementing a counter within a ThreadLocal, so I focus on the time to do the more expensive aggregate (involving a synchronized method to update the MutableRate metric). First I run a test with only a single thread updating metrics, then do the full 200 threads under a few different conditions: turning on and off aggregation (to get a baseline figure of performance with many threads running), and including a 1-millisecond sleep between operations (to emulate slightly less pessimistic conditions of lock contention). Each thread does 10,000 aggregations and I measure the time per operation; over 10 trials I got: 10000 aggregation per thread over 100 trials Test Average Time (ns) Std Dev (ns) Single Thread 3107 606 No Agg, No Wait 551 551 Agg, No Wait 235850 24059 No Agg, Wait 1065525 625 Agg, Wait 1158477 8743 So it seems that even under highly contested conditions an aggregation adds ~100-200 microseconds to the execution path, and without contention only ~3-4 microseconds. Given that a typical aggregation period would be the same as the metrics collection interval, say 10-60 seconds, this seems reasonable for a disabled-by-default feature.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for the patch! The structure looks good overall. A few comments:

          1. Since block reporting is a major factor of write locking, I think we should add the new unlock API to BlockManager. We are not removing any existing APIs so compatibility shouldn't be a concern. Due the size of the current patch we can probably do that in a separate JIRA though.
          2. Interesting that we are almost aligned with how audit logs are recorded. On the one hand this confirms that there's probably no easier way to get the method name. On the other hand we should construct a String variable that can be reused by audit logging and this locking metrics logic. This avoids one temporary variable for each method call.
          3. Have you considered integrating the Metrics logic into NameNodeMetrics?
          Show
          zhz Zhe Zhang added a comment - Thanks Erik for the patch! The structure looks good overall. A few comments: Since block reporting is a major factor of write locking, I think we should add the new unlock API to BlockManager . We are not removing any existing APIs so compatibility shouldn't be a concern. Due the size of the current patch we can probably do that in a separate JIRA though. Interesting that we are almost aligned with how audit logs are recorded. On the one hand this confirms that there's probably no easier way to get the method name. On the other hand we should construct a String variable that can be reused by audit logging and this locking metrics logic. This avoids one temporary variable for each method call. Have you considered integrating the Metrics logic into NameNodeMetrics ?
          Hide
          xkrogen Erik Krogen added a comment -

          Thanks for the review, Zhe Zhang.

          1. I agree. Created HDFS-11021 to track.
          2. Used a final String variable for each operation whose name is used more than once.
          3. Per our offline discussion the best place for these metrics to live is actually FSNamesystem's metrics. Thank you for bringing up the issue.

          Attaching v001 patch with the above changes and some bug fixes.

          Show
          xkrogen Erik Krogen added a comment - Thanks for the review, Zhe Zhang . 1. I agree. Created HDFS-11021 to track. 2. Used a final String variable for each operation whose name is used more than once. 3. Per our offline discussion the best place for these metrics to live is actually FSNamesystem 's metrics. Thank you for bringing up the issue. Attaching v001 patch with the above changes and some bug fixes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 58s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 830 unchanged - 0 fixed = 836 total (was 830)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 40s the patch passed
          -1 unit 77m 35s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          98m 2s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Should org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock$LocalMetric be a static inner class? At FSNamesystemLock.java:inner class? At FSNamesystemLock.java:[lines 359-376]
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833771/HDFS-10872.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ae23194037c7 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ed9fcbe
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17183/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17183/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17183/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17183/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17183/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 58s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 830 unchanged - 0 fixed = 836 total (was 830) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 40s the patch passed -1 unit 77m 35s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 98m 2s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Should org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock$LocalMetric be a static inner class? At FSNamesystemLock.java:inner class? At FSNamesystemLock.java: [lines 359-376] Failed junit tests hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833771/HDFS-10872.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ae23194037c7 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed9fcbe Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17183/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17183/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/17183/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17183/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17183/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Attaching v002 patch addressing findbugs and checkstyle. Unit test failure doesn't seem related; passes fine locally.

          Show
          xkrogen Erik Krogen added a comment - Attaching v002 patch addressing findbugs and checkstyle. Unit test failure doesn't seem related; passes fine locally.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for the updated patch. I think we are pretty close.

          About the main change in FSNamesystemLock

          1. If a thread lives shorter than the configured metricAggregationInterval, are we gonna lose its locking metrics? In reality it's probably not a big concern since RPC handler threads are reused. But putting a comment here for more thoughts.
          2. Selecting a real default aggregation interval is not easy. Maybe we should document it in hdfs-default.xml. Alternatively we can have 2 config knobs, one binary and one integer.
          3. Agreed that the overhead with the current implementation is pretty small (say with 10 sec interval). As a follow-on optimizaiton (depending on experience from production deployment) maybe we can consider a combination of hard-deadline and opportunistic try-and-backoff. E.g. at a higher frequency than metricAggregationInterval we can try the lock for opHoldtimeMetrics; if lock is free, dump the metrics, otherwise try lock in some more time.

          A few minors about naming:

          1. The below mapping merges the yield() locking time into the contentSummary category. It looks a reasonable approximation to me. But more opinions would be helpful.
            @@ -115,7 +115,7 @@ public boolean yield() {
                 // unlock
                 dir.readUnlock();
            -    fsn.readUnlock();
            +    fsn.readUnlock("contentSummary");
            
          2. getBlockLocations currently appears as open in audit logging. It could be an existing bug. But since it has been audit logged that way it doesn't make sense to change here. I'll create a separate JIRA to discuss.
          3. writeUnlock("clearCorruptLazyPersistFile"); should be "clearCorruptLazyPersistFiles"
          4. Maybe the checkLease category should be leasesMonitor? There's a one-off checkLease() method in FSN
          5. The one in NamenodeFsck#getBlockLocations, maybe we should use "fsckGetBlockLocations" to differentiate from regular getBlockLocations RPC call?
          Show
          zhz Zhe Zhang added a comment - Thanks Erik for the updated patch. I think we are pretty close. About the main change in FSNamesystemLock If a thread lives shorter than the configured metricAggregationInterval , are we gonna lose its locking metrics? In reality it's probably not a big concern since RPC handler threads are reused. But putting a comment here for more thoughts. Selecting a real default aggregation interval is not easy. Maybe we should document it in hdfs-default.xml . Alternatively we can have 2 config knobs, one binary and one integer. Agreed that the overhead with the current implementation is pretty small (say with 10 sec interval). As a follow-on optimizaiton (depending on experience from production deployment) maybe we can consider a combination of hard-deadline and opportunistic try-and-backoff. E.g. at a higher frequency than metricAggregationInterval we can try the lock for opHoldtimeMetrics ; if lock is free, dump the metrics, otherwise try lock in some more time. A few minors about naming: The below mapping merges the yield() locking time into the contentSummary category. It looks a reasonable approximation to me. But more opinions would be helpful. @@ -115,7 +115,7 @@ public boolean yield() { // unlock dir.readUnlock(); - fsn.readUnlock(); + fsn.readUnlock( "contentSummary" ); getBlockLocations currently appears as open in audit logging. It could be an existing bug. But since it has been audit logged that way it doesn't make sense to change here. I'll create a separate JIRA to discuss. writeUnlock("clearCorruptLazyPersistFile"); should be "clearCorruptLazyPersistFiles" Maybe the checkLease category should be leasesMonitor ? There's a one-off checkLease() method in FSN The one in NamenodeFsck#getBlockLocations , maybe we should use "fsckGetBlockLocations" to differentiate from regular getBlockLocations RPC call?
          Hide
          xkrogen Erik Krogen added a comment -

          Zhe Zhang thanks for the detailed review!

          1. Yes, if it lives shorter than the aggregation interval, its metrics will be lost. I was thinking the same that the threads are re-used so very few metrics should be lost, and also from within the FSNamesystemLock there wouldn't be any easy way to detect that the thread is exiting to flush its metrics. I think this is worth documenting though; I've added some discussion on the FSNamesystemLock Javadoc.
          2. Good point. I've added some guidance about a good default in hdfs-default.xml.
          3. This seems like a pretty interesting idea. It could be pretty simple / little implementation effort, but I think there is some question of if it will actually be helpful. I can code something up for this patch or leave it... Let me know what you think.

          1. Right. That yield() is only called as part of the contentSummary operation, just quickly releasing and reacquiring the lock which contentSummary originally held. The yield could make a single contentSummary appear as multiple operations instead of a single one, but the total time should still be accurate.
          2. For any operation that already had an audit log name I kept that for consistency. Agreed that it doesn't make sense to change here.
          3. Thanks, fixed.
          4. Yes, agreed. Not sure why I originally went with checkLease. I've changed it to leaseManager to be consistent with the name of the class.
          5. Agreed, changed. Good catch.

          Attaching v003 patch with changes discussed.

          Show
          xkrogen Erik Krogen added a comment - Zhe Zhang thanks for the detailed review! 1. Yes, if it lives shorter than the aggregation interval, its metrics will be lost. I was thinking the same that the threads are re-used so very few metrics should be lost, and also from within the FSNamesystemLock there wouldn't be any easy way to detect that the thread is exiting to flush its metrics. I think this is worth documenting though; I've added some discussion on the FSNamesystemLock Javadoc. 2. Good point. I've added some guidance about a good default in hdfs-default.xml . 3. This seems like a pretty interesting idea. It could be pretty simple / little implementation effort, but I think there is some question of if it will actually be helpful. I can code something up for this patch or leave it... Let me know what you think. 1. Right. That yield() is only called as part of the contentSummary operation, just quickly releasing and reacquiring the lock which contentSummary originally held. The yield could make a single contentSummary appear as multiple operations instead of a single one, but the total time should still be accurate. 2. For any operation that already had an audit log name I kept that for consistency. Agreed that it doesn't make sense to change here. 3. Thanks, fixed. 4. Yes, agreed. Not sure why I originally went with checkLease . I've changed it to leaseManager to be consistent with the name of the class. 5. Agreed, changed. Good catch. Attaching v003 patch with changes discussed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 44s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 38s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 44s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 46s the patch passed
          +1 javac 0m 46s the patch passed
          -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 830 unchanged - 0 fixed = 831 total (was 830)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 50s the patch passed
          +1 javadoc 0m 39s the patch passed
          -1 unit 77m 48s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          99m 25s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833997/HDFS-10872.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 054dc15365c5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b733a6f
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17205/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17205/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17205/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17205/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 44s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 830 unchanged - 0 fixed = 831 total (was 830) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 77m 48s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 99m 25s Reason Tests Failed junit tests hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833997/HDFS-10872.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 054dc15365c5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b733a6f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17205/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17205/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17205/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17205/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik! The v3 patch LGTM. +1 pending fixing the long line checkstyle issue.

          * {@link org.apache.hadoop.hdfs.DFSConfigKeys#DFS_NAMENODE_LOCK_METRIC_AGG_INTERVAL_MS_KEY}
          

          I think we should leave the try-and-backoff as a potential optimization.

          I will hold off committing the patch until EOD Thursday to collect more feedbacks. Kihwal Lee Do you think the patch addresses your use cases ? Andrew Wang Do you think the aggregation method addresses your concern on performance (see benchmark above)? Mingliang Liu Any comments on the form of output?

          Show
          zhz Zhe Zhang added a comment - Thanks Erik! The v3 patch LGTM. +1 pending fixing the long line checkstyle issue. * {@link org.apache.hadoop.hdfs.DFSConfigKeys#DFS_NAMENODE_LOCK_METRIC_AGG_INTERVAL_MS_KEY} I think we should leave the try-and-backoff as a potential optimization. I will hold off committing the patch until EOD Thursday to collect more feedbacks. Kihwal Lee Do you think the patch addresses your use cases ? Andrew Wang Do you think the aggregation method addresses your concern on performance (see benchmark above)? Mingliang Liu Any comments on the form of output?
          Hide
          xkrogen Erik Krogen added a comment -

          Attaching v004 to fix checkstyle.

          Waiting sounds good. Thanks Zhe.

          Show
          xkrogen Erik Krogen added a comment - Attaching v004 to fix checkstyle. Waiting sounds good. Thanks Zhe.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this folks, and sorry for coming to this review late. I have a few comments looking at this patch:

          • I'd prefer we use try/finally to do the lock/unlocks in aggregateMetrics for safety.
          • We're doing the thread-local increment and aggregate outside of the metrics system right now. Did we look into making the metrics classes themselves multi-threading friendly? Then we could more easily reuse this code in other places.
          • Related, the aggregation right now is done periodically on increment. I think what we'd really like is aggregate-on-read. The metrics classes support this via setChanged and snapshot. Otherwise, occasionally one operation is getting hit with the penalty of aggregating all the metrics, and possibly for no reason if it's not being read.

          I might have mentioned this before, but the LongAdder code in JDK8 is basically what I'm thinking of, and the performance is quite good. The license is friendly, so we could copy-paste this into Hadoop for the branch-2's, and use the built-in JDK8 version for trunk.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this folks, and sorry for coming to this review late. I have a few comments looking at this patch: I'd prefer we use try/finally to do the lock/unlocks in aggregateMetrics for safety. We're doing the thread-local increment and aggregate outside of the metrics system right now. Did we look into making the metrics classes themselves multi-threading friendly? Then we could more easily reuse this code in other places. Related, the aggregation right now is done periodically on increment. I think what we'd really like is aggregate-on-read. The metrics classes support this via setChanged and snapshot . Otherwise, occasionally one operation is getting hit with the penalty of aggregating all the metrics, and possibly for no reason if it's not being read. I might have mentioned this before, but the LongAdder code in JDK8 is basically what I'm thinking of, and the performance is quite good. The license is friendly, so we could copy-paste this into Hadoop for the branch-2's, and use the built-in JDK8 version for trunk.
          Hide
          xkrogen Erik Krogen added a comment -

          Andrew Wang, thanks for the comment.

          I actually did not think to modify the metrics classes rather than working around their current limitations but this is definitely good to consider. We are specifically interested in the MutableRate metric here, which is currently synchronized since it requires updating of multiple values atomically. For something like MutableCounter a LongAdder would be great, but I am wondering if we can make it apply here? I particularly note the Javadoc which says, of resets,

          Because this method is intrinsically racy, it should only be used when it is known that no threads are concurrently updating.

          But we would need the ability to do resets for MutableRate. I suppose we could potentially read lock around a normal increment of a LongAdder and write lock upon snapshotting; or, probably more performant, use an AtomicReference to store the LongAdder and create a new one when we snapshot rather than attempting to reset. Any thoughts on that?

          Show
          xkrogen Erik Krogen added a comment - Andrew Wang , thanks for the comment. I actually did not think to modify the metrics classes rather than working around their current limitations but this is definitely good to consider. We are specifically interested in the MutableRate metric here, which is currently synchronized since it requires updating of multiple values atomically. For something like MutableCounter a LongAdder would be great, but I am wondering if we can make it apply here? I particularly note the Javadoc which says, of resets, Because this method is intrinsically racy, it should only be used when it is known that no threads are concurrently updating. But we would need the ability to do resets for MutableRate. I suppose we could potentially read lock around a normal increment of a LongAdder and write lock upon snapshotting; or, probably more performant, use an AtomicReference to store the LongAdder and create a new one when we snapshot rather than attempting to reset. Any thoughts on that?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 40s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 830 unchanged - 0 fixed = 831 total (was 830)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 36s the patch passed
          -1 unit 67m 28s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          85m 53s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.hdfs.server.datanode.TestDataNodeLifeline
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834065/HDFS-10872.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux e0ae13c476f1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c62ae71
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17210/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17210/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17210/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17210/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 40s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 830 unchanged - 0 fixed = 831 total (was 830) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 36s the patch passed -1 unit 67m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 85m 53s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834065/HDFS-10872.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e0ae13c476f1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c62ae71 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17210/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17210/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17210/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17210/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the discussion Andrew and Erik.

          I agree that moving the multi-threading logic to Metrics classes is a good idea. I'm actually pretty curious how that will improve the performance of existing RPC metrics collection; it seems pretty expensive. We should probably do that as a prerequisite and then rebase this patch.

          Using LongAdder is an interesting idea. Andrew Wang This is actually the first time I learn about it Are we aware of any production usage of this library? I took a look at the test coverage; it seems a little weak considering the complexity of the main logic. We can probably consider adding some more unit tests in Hadoop to cover cases we are most interested in.

          I expect that a few minutes after the NN starts, the set of operations (keys) in the opHoldtimeMetrics map is stablized. Afterwards, each entry is essentially a counter to be incremented. So on the high level I think we should be able to make it work with LongAdder. Agreed with Erik Krogen that snapshotting/resetting is probably the trickiest part. Creating a new LongAdder in snapshot sounds reseaonable to me.

          Show
          zhz Zhe Zhang added a comment - Thanks for the discussion Andrew and Erik. I agree that moving the multi-threading logic to Metrics classes is a good idea. I'm actually pretty curious how that will improve the performance of existing RPC metrics collection; it seems pretty expensive. We should probably do that as a prerequisite and then rebase this patch. Using LongAdder is an interesting idea. Andrew Wang This is actually the first time I learn about it Are we aware of any production usage of this library? I took a look at the test coverage; it seems a little weak considering the complexity of the main logic. We can probably consider adding some more unit tests in Hadoop to cover cases we are most interested in. I expect that a few minutes after the NN starts, the set of operations (keys) in the opHoldtimeMetrics map is stablized. Afterwards, each entry is essentially a counter to be incremented. So on the high level I think we should be able to make it work with LongAdder . Agreed with Erik Krogen that snapshotting/resetting is probably the trickiest part. Creating a new LongAdder in snapshot sounds reseaonable to me.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the replies, Erik, Zhe, inline:

          Are we aware of any production usage of this library?

          The popular codahale/Dropwizard metrics library pulls in LongAdder in the manner I suggested, so it might be easiest to add that as a dependency. I'm pretty sure we already have it floating around in Hadoop somewhere. The quality of JDK code is also well regarded.

          For something like MutableCounter a LongAdder would be great, but I am wondering if we can make it apply here?

          Looking at EWMA in codahale, they reset a LongAdder without additional synchronization with updates. Metrics don't need to be super precise, so as long as there isn't a disastrous concurrent failure mode, this is okay.

          Unless we really need the additional bits of MutableRate, the simplest thing might be to wrap the Dropwizard metrics classes like Histogram and EWMA to fit in the Hadoop metrics system.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the replies, Erik, Zhe, inline: Are we aware of any production usage of this library? The popular codahale/Dropwizard metrics library pulls in LongAdder in the manner I suggested, so it might be easiest to add that as a dependency. I'm pretty sure we already have it floating around in Hadoop somewhere. The quality of JDK code is also well regarded. For something like MutableCounter a LongAdder would be great, but I am wondering if we can make it apply here? Looking at EWMA in codahale , they reset a LongAdder without additional synchronization with updates. Metrics don't need to be super precise, so as long as there isn't a disastrous concurrent failure mode, this is okay. Unless we really need the additional bits of MutableRate, the simplest thing might be to wrap the Dropwizard metrics classes like Histogram and EWMA to fit in the Hadoop metrics system.
          Hide
          xkrogen Erik Krogen added a comment -

          Andrew Wang I did some profiling and it seems that the performance hit to keeping a consistent view and using an atomic reference is pretty negligible. Benchmark results below, running each benchmark for 20 seconds with 100 competing reader threads incrementing the counter as quickly as they can and a varying snapshot interval:

          Benchmark                          (metricType)  (snapshotIntervalMs)  Mode  Cnt      Score     Error  Units
          MetricBenchmark.increment          Synchronized                   100  avgt   10  10802.943 ± 366.821  ns/op
          MetricBenchmark.increment          Synchronized                  1000  avgt   10   8991.085 ± 224.738  ns/op
          MetricBenchmark.increment          Synchronized                  5000  avgt   10   9780.644 ± 258.482  ns/op
          MetricBenchmark.increment          Synchronized                 10000  avgt   10   8864.450 ± 239.349  ns/op
          MetricBenchmark.increment           RWLockAdder                   100  avgt   10  31546.824 ± 203.405  ns/op
          MetricBenchmark.increment           RWLockAdder                  1000  avgt   10  31350.341 ± 156.414  ns/op
          MetricBenchmark.increment           RWLockAdder                  5000  avgt   10  33338.370 ± 234.421  ns/op
          MetricBenchmark.increment           RWLockAdder                 10000  avgt   10  31057.551 ± 211.002  ns/op
          MetricBenchmark.increment  AtomicReferenceAdder                   100  avgt   10    172.594 ±   1.478  ns/op
          MetricBenchmark.increment  AtomicReferenceAdder                  1000  avgt   10    171.760 ±   0.647  ns/op
          MetricBenchmark.increment  AtomicReferenceAdder                  5000  avgt   10    172.942 ±   2.991  ns/op
          MetricBenchmark.increment  AtomicReferenceAdder                 10000  avgt   10    186.659 ±  23.906  ns/op
          MetricBenchmark.increment      UnprotectedAdder                   100  avgt   10    202.249 ±   0.371  ns/op
          MetricBenchmark.increment      UnprotectedAdder                  1000  avgt   10    166.536 ±   0.273  ns/op
          MetricBenchmark.increment      UnprotectedAdder                  5000  avgt   10    179.672 ±   0.335  ns/op
          MetricBenchmark.increment      UnprotectedAdder                 10000  avgt   10    168.062 ±   2.149  ns/op
          

          Synchronized is basically the current approach (a synchronized method), RWLockAdder uses a ReadWriteLock to protect a LongAdder, AtomicReferenceAdder uses an AtomicReference to protect a LongAdder, and UnprotectedAdder simply uses a LongAdder with an unprotected sumThenReset. It seems based off of this that using an AtomicReference to protect a LongAdder (or, an object containing a small number of LongAdders for e.g. MutableRate which needs to maintain multiple counters) would provide a consistent snapshot and be much more performant than our current solution.

          Thoughts?

          Show
          xkrogen Erik Krogen added a comment - Andrew Wang I did some profiling and it seems that the performance hit to keeping a consistent view and using an atomic reference is pretty negligible. Benchmark results below, running each benchmark for 20 seconds with 100 competing reader threads incrementing the counter as quickly as they can and a varying snapshot interval: Benchmark (metricType) (snapshotIntervalMs) Mode Cnt Score Error Units MetricBenchmark.increment Synchronized 100 avgt 10 10802.943 ± 366.821 ns/op MetricBenchmark.increment Synchronized 1000 avgt 10 8991.085 ± 224.738 ns/op MetricBenchmark.increment Synchronized 5000 avgt 10 9780.644 ± 258.482 ns/op MetricBenchmark.increment Synchronized 10000 avgt 10 8864.450 ± 239.349 ns/op MetricBenchmark.increment RWLockAdder 100 avgt 10 31546.824 ± 203.405 ns/op MetricBenchmark.increment RWLockAdder 1000 avgt 10 31350.341 ± 156.414 ns/op MetricBenchmark.increment RWLockAdder 5000 avgt 10 33338.370 ± 234.421 ns/op MetricBenchmark.increment RWLockAdder 10000 avgt 10 31057.551 ± 211.002 ns/op MetricBenchmark.increment AtomicReferenceAdder 100 avgt 10 172.594 ± 1.478 ns/op MetricBenchmark.increment AtomicReferenceAdder 1000 avgt 10 171.760 ± 0.647 ns/op MetricBenchmark.increment AtomicReferenceAdder 5000 avgt 10 172.942 ± 2.991 ns/op MetricBenchmark.increment AtomicReferenceAdder 10000 avgt 10 186.659 ± 23.906 ns/op MetricBenchmark.increment UnprotectedAdder 100 avgt 10 202.249 ± 0.371 ns/op MetricBenchmark.increment UnprotectedAdder 1000 avgt 10 166.536 ± 0.273 ns/op MetricBenchmark.increment UnprotectedAdder 5000 avgt 10 179.672 ± 0.335 ns/op MetricBenchmark.increment UnprotectedAdder 10000 avgt 10 168.062 ± 2.149 ns/op Synchronized is basically the current approach (a synchronized method), RWLockAdder uses a ReadWriteLock to protect a LongAdder , AtomicReferenceAdder uses an AtomicReference to protect a LongAdder , and UnprotectedAdder simply uses a LongAdder with an unprotected sumThenReset . It seems based off of this that using an AtomicReference to protect a LongAdder (or, an object containing a small number of LongAdders for e.g. MutableRate which needs to maintain multiple counters) would provide a consistent snapshot and be much more performant than our current solution. Thoughts?
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the benchmark Erik Krogen! Results look very interesting. I think it's time to move the discussion to a separate metrics JIRA. Just created HADOOP-13747.

          Show
          zhz Zhe Zhang added a comment - Thanks for the benchmark Erik Krogen ! Results look very interesting. I think it's time to move the discussion to a separate metrics JIRA. Just created HADOOP-13747 .
          Hide
          andrew.wang Andrew Wang added a comment -

          This looks great, thanks Erik! I was a bit worried since LongAdder needs to expand the hashtable as it warms up, but the hit looks marginal. Let's pursue this further in HADOOP-13747.

          Show
          andrew.wang Andrew Wang added a comment - This looks great, thanks Erik! I was a bit worried since LongAdder needs to expand the hashtable as it warms up, but the hit looks marginal. Let's pursue this further in HADOOP-13747 .
          Hide
          xkrogen Erik Krogen added a comment -

          Now that HADOOP-13782 has provided us with a metrics class which provides fast concurrent access to MutableRate metrics (MutableRatesWithAggregation), the patch has been refactored to make use of this. This required a minor modification to MetricsRegistry to export a newMutableRatesWithAggregation method to create a new metrics object in FSNamesystem to be able to pass into FSNamesystemLock. This is necessary since we want the lock hold metrics to be emitted within the FSNamesystem metrics registry but they are generated within FSNamesystemLock, so we pass the metric object down into the lock for modification.

          Attaching v005 patch.

          Show
          xkrogen Erik Krogen added a comment - Now that HADOOP-13782 has provided us with a metrics class which provides fast concurrent access to MutableRate metrics ( MutableRatesWithAggregation ), the patch has been refactored to make use of this. This required a minor modification to MetricsRegistry to export a newMutableRatesWithAggregation method to create a new metrics object in FSNamesystem to be able to pass into FSNamesystemLock . This is necessary since we want the lock hold metrics to be emitted within the FSNamesystem metrics registry but they are generated within FSNamesystemLock , so we pass the metric object down into the lock for modification. Attaching v005 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 23s Maven dependency ordering for branch
          +1 mvninstall 7m 21s trunk passed
          +1 compile 12m 0s trunk passed
          +1 checkstyle 1m 48s trunk passed
          +1 mvnsite 2m 17s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          +1 findbugs 3m 34s trunk passed
          +1 javadoc 1m 41s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 44s the patch passed
          +1 compile 10m 38s the patch passed
          +1 javac 10m 38s the patch passed
          -0 checkstyle 1m 49s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862)
          +1 mvnsite 2m 6s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 3m 41s the patch passed
          +1 javadoc 1m 41s the patch passed
          +1 unit 9m 7s hadoop-common in the patch passed.
          -1 unit 67m 41s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 49s The patch does not generate ASF License warnings.
          131m 44s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.server.namenode.TestMetaSave



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838271/HDFS-10872.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ec833f598ec1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 71adf44
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17494/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17494/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17494/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17494/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 7m 21s trunk passed +1 compile 12m 0s trunk passed +1 checkstyle 1m 48s trunk passed +1 mvnsite 2m 17s trunk passed +1 mvneclipse 0m 44s trunk passed +1 findbugs 3m 34s trunk passed +1 javadoc 1m 41s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 44s the patch passed +1 compile 10m 38s the patch passed +1 javac 10m 38s the patch passed -0 checkstyle 1m 49s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862) +1 mvnsite 2m 6s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 41s the patch passed +1 javadoc 1m 41s the patch passed +1 unit 9m 7s hadoop-common in the patch passed. -1 unit 67m 41s hadoop-hdfs in the patch failed. +1 asflicense 0m 49s The patch does not generate ASF License warnings. 131m 44s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.server.namenode.TestMetaSave Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838271/HDFS-10872.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ec833f598ec1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 71adf44 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17494/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17494/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17494/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17494/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment - - edited

          Test failures don't seem related and pass fine locally.

          Checkstyle is complaining about an import only used within a Javadoc (marking as unused) - changed it slightly so that checkstyle won't complain. Attaching v006 patch.

          Show
          xkrogen Erik Krogen added a comment - - edited Test failures don't seem related and pass fine locally. Checkstyle is complaining about an import only used within a Javadoc (marking as unused) - changed it slightly so that checkstyle won't complain. Attaching v006 patch.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for the patch! LGTM overall. I think the added newMutableRatesWithAggregation constructor is reasonable. Some minors:

          1. Below change doesn't seem necessary
            -  private final MetricsRegistry registry = new MetricsRegistry("FSNamesystem");
            +  private final MetricsRegistry registry =
            +      new MetricsRegistry("FSNamesystem").setContext("dfs");
            
          2. We could have the @link have its own line, right now the link is broken:
             * If metrics are enabled by setting {@link org.apache.hadoop.hdfs.DFSConfigKeys
             * #DFS_NAMENODE_LOCK_DETAILED_METRICS_KEY} to be true, metrics will be emitted
             
          3. Follow-on: maybe we can consider consolidating the code in readUnlock and writeUnlock

          Andrew Wang Any further comments? Thanks.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik for the patch! LGTM overall. I think the added newMutableRatesWithAggregation constructor is reasonable. Some minors: Below change doesn't seem necessary - private final MetricsRegistry registry = new MetricsRegistry( "FSNamesystem" ); + private final MetricsRegistry registry = + new MetricsRegistry( "FSNamesystem" ).setContext( "dfs" ); We could have the @link have its own line, right now the link is broken: * If metrics are enabled by setting {@link org.apache.hadoop.hdfs.DFSConfigKeys * #DFS_NAMENODE_LOCK_DETAILED_METRICS_KEY} to be true , metrics will be emitted Follow-on: maybe we can consider consolidating the code in readUnlock and writeUnlock Andrew Wang Any further comments? Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 38s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 2m 5s Maven dependency ordering for branch
          +1 mvninstall 12m 15s trunk passed
          -1 compile 7m 7s root in trunk failed.
          +1 checkstyle 1m 47s trunk passed
          +1 mvnsite 2m 9s trunk passed
          +1 mvneclipse 2m 0s trunk passed
          +1 findbugs 8m 32s trunk passed
          +1 javadoc 4m 56s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          -1 compile 5m 39s root in the patch failed.
          -1 javac 5m 39s root in the patch failed.
          -0 checkstyle 1m 45s root: The patch generated 1 new + 862 unchanged - 0 fixed = 863 total (was 862)
          +1 mvnsite 2m 1s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 3m 28s the patch passed
          +1 javadoc 1m 35s the patch passed
          +1 unit 7m 21s hadoop-common in the patch passed.
          -1 unit 55m 31s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          123m 45s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.TestDFSAdmin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838401/HDFS-10872.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4cd2b8c865a8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89354f0
          Default Java 1.8.0_101
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/branch-compile-root.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17509/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17509/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 38s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 2m 5s Maven dependency ordering for branch +1 mvninstall 12m 15s trunk passed -1 compile 7m 7s root in trunk failed. +1 checkstyle 1m 47s trunk passed +1 mvnsite 2m 9s trunk passed +1 mvneclipse 2m 0s trunk passed +1 findbugs 8m 32s trunk passed +1 javadoc 4m 56s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed -1 compile 5m 39s root in the patch failed. -1 javac 5m 39s root in the patch failed. -0 checkstyle 1m 45s root: The patch generated 1 new + 862 unchanged - 0 fixed = 863 total (was 862) +1 mvnsite 2m 1s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 28s the patch passed +1 javadoc 1m 35s the patch passed +1 unit 7m 21s hadoop-common in the patch passed. -1 unit 55m 31s hadoop-hdfs in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 123m 45s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDFSAdmin Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838401/HDFS-10872.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4cd2b8c865a8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89354f0 Default Java 1.8.0_101 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17509/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17509/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Zhe Zhang, thanks for the review.

          (1) Yes, you are right, that is unnecessary.
          (2) Fixed. I'm surprised my IDE didn't warn me of this.
          (3) Agreed that this could be a good follow-up, there is a lot of duplicated logic. Could be good to do something to consolidate with HADOOP-13702 as well.

          Attaching v007 patch.

          Show
          xkrogen Erik Krogen added a comment - Zhe Zhang , thanks for the review. (1) Yes, you are right, that is unnecessary. (2) Fixed. I'm surprised my IDE didn't warn me of this. (3) Agreed that this could be a good follow-up, there is a lot of duplicated logic. Could be good to do something to consolidate with HADOOP-13702 as well. Attaching v007 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 8m 58s trunk passed
          -1 compile 7m 21s root in trunk failed.
          +1 checkstyle 1m 48s trunk passed
          +1 mvnsite 2m 5s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 3m 21s trunk passed
          +1 javadoc 1m 53s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 33s the patch passed
          -1 compile 6m 14s root in the patch failed.
          -1 javac 6m 14s root in the patch failed.
          -0 checkstyle 1m 49s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862)
          +1 mvnsite 2m 9s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 36s the patch passed
          +1 javadoc 1m 42s the patch passed
          +1 unit 8m 2s hadoop-common in the patch passed.
          -1 unit 60m 35s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          115m 27s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.tools.TestDFSAdmin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838420/HDFS-10872.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux cfea7118d697 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89354f0
          Default Java 1.8.0_101
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/branch-compile-root.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17513/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17513/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 58s trunk passed -1 compile 7m 21s root in trunk failed. +1 checkstyle 1m 48s trunk passed +1 mvnsite 2m 5s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 3m 21s trunk passed +1 javadoc 1m 53s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 33s the patch passed -1 compile 6m 14s root in the patch failed. -1 javac 6m 14s root in the patch failed. -0 checkstyle 1m 49s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862) +1 mvnsite 2m 9s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 36s the patch passed +1 javadoc 1m 42s the patch passed +1 unit 8m 2s hadoop-common in the patch passed. -1 unit 60m 35s hadoop-hdfs in the patch failed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 115m 27s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.tools.TestDFSAdmin Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838420/HDFS-10872.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux cfea7118d697 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89354f0 Default Java 1.8.0_101 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17513/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17513/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik, +1 on v7 patch pending a successful Jenkins run (I just re-triggerred).

          Will also wait for a day before committing so others can comment.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik, +1 on v7 patch pending a successful Jenkins run (I just re-triggerred). Will also wait for a day before committing so others can comment.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 24s trunk passed
          -1 compile 7m 35s root in trunk failed.
          +1 checkstyle 1m 47s trunk passed
          +1 mvnsite 2m 10s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 3m 19s trunk passed
          +1 javadoc 1m 40s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          -1 compile 5m 59s root in the patch failed.
          -1 javac 5m 59s root in the patch failed.
          -0 checkstyle 1m 44s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862)
          +1 mvnsite 2m 7s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 35s the patch passed
          +1 javadoc 1m 38s the patch passed
          +1 unit 7m 44s hadoop-common in the patch passed.
          -1 unit 57m 20s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          109m 51s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.TestDFSAdmin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838420/HDFS-10872.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 8c888dce6de0 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89354f0
          Default Java 1.8.0_101
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/branch-compile-root.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17517/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17517/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 24s trunk passed -1 compile 7m 35s root in trunk failed. +1 checkstyle 1m 47s trunk passed +1 mvnsite 2m 10s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 3m 19s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed -1 compile 5m 59s root in the patch failed. -1 javac 5m 59s root in the patch failed. -0 checkstyle 1m 44s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862) +1 mvnsite 2m 7s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 35s the patch passed +1 javadoc 1m 38s the patch passed +1 unit 7m 44s hadoop-common in the patch passed. -1 unit 57m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 109m 51s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDFSAdmin Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838420/HDFS-10872.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8c888dce6de0 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89354f0 Default Java 1.8.0_101 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17517/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17517/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment - - edited

          Attaching v008 patch which includes the changes within YARN-5868 (fixing an issue with npm during the build process, entirely unrelated to this ticket) just to be able to get a clean Jenkins build. Patch is otherwise identical to v007.

          Show
          xkrogen Erik Krogen added a comment - - edited Attaching v008 patch which includes the changes within YARN-5868 (fixing an issue with npm during the build process, entirely unrelated to this ticket) just to be able to get a clean Jenkins build. Patch is otherwise identical to v007.
          Hide
          xkrogen Erik Krogen added a comment -

          Looks like YARN-5868 went through faster than expected, reuploading v007 patch as v009 to trigger Jenkins.

          Show
          xkrogen Erik Krogen added a comment - Looks like YARN-5868 went through faster than expected, reuploading v007 patch as v009 to trigger Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 6m 45s trunk passed
          +1 compile 10m 14s trunk passed
          +1 checkstyle 1m 45s trunk passed
          +1 mvnsite 2m 16s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 3m 32s trunk passed
          +1 javadoc 1m 40s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 43s the patch passed
          +1 compile 10m 55s the patch passed
          +1 javac 10m 55s the patch passed
          -0 checkstyle 1m 44s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862)
          +1 mvnsite 2m 2s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 3m 31s the patch passed
          +1 javadoc 1m 30s the patch passed
          +1 unit 8m 29s hadoop-common in the patch passed.
          -1 unit 58m 10s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          117m 53s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.tools.TestDelegationTokenFetcher
            hadoop.hdfs.TestRenameWhileOpen



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838602/HDFS-10872.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 621f3278cf6d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5c61ad2
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17533/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17533/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17533/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17533/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 10m 14s trunk passed +1 checkstyle 1m 45s trunk passed +1 mvnsite 2m 16s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 3m 32s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 43s the patch passed +1 compile 10m 55s the patch passed +1 javac 10m 55s the patch passed -0 checkstyle 1m 44s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862) +1 mvnsite 2m 2s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 31s the patch passed +1 javadoc 1m 30s the patch passed +1 unit 8m 29s hadoop-common in the patch passed. -1 unit 58m 10s hadoop-hdfs in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 117m 53s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838602/HDFS-10872.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 621f3278cf6d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5c61ad2 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17533/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17533/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17533/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17533/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          TestDFSAdmin is a known issue tracked in HDFS-11122. TestRenameWhileOpen/TestDelegationTokenFetcher pass locally and seem unrelated.

          Show
          xkrogen Erik Krogen added a comment - TestDFSAdmin is a known issue tracked in HDFS-11122 . TestRenameWhileOpen/TestDelegationTokenFetcher pass locally and seem unrelated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Tan, Wangda for helping fix the npm error. Thanks Erik for verifying pre-commit.

          +1 on the patch, will work on trunk~branch-2.7 commit now.

          Show
          zhz Zhe Zhang added a comment - Thanks Tan, Wangda for helping fix the npm error. Thanks Erik for verifying pre-commit. +1 on the patch, will work on trunk~branch-2.7 commit now.
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          Show
          liuml07 Mingliang Liu added a comment - +1
          Hide
          zhz Zhe Zhang added a comment -

          Sorry, just realized an issue with the names of the metrics, can we more explicitly specify those metrics are for locks?

          Show
          zhz Zhe Zhang added a comment - Sorry, just realized an issue with the names of the metrics, can we more explicitly specify those metrics are for locks?
          Hide
          liuml07 Mingliang Liu added a comment -

          That's a good point. We can add prefix (e.g. FSNLock) in FSNamesystemLock#addMetric().

          Show
          liuml07 Mingliang Liu added a comment - That's a good point. We can add prefix (e.g. FSNLock) in FSNamesystemLock#addMetric() .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 23 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 9m 22s trunk passed
          -1 compile 9m 10s root in trunk failed.
          +1 checkstyle 2m 31s trunk passed
          +1 mvnsite 11m 32s trunk passed
          +1 mvneclipse 4m 34s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-cloud-storage-project hadoop-cloud-storage-project/hadoop-cloud-storage
          -1 findbugs 0m 18s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          +1 javadoc 5m 41s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 15m 30s the patch passed
          -1 compile 5m 57s root in the patch failed.
          -1 cc 5m 57s root in the patch failed.
          -1 javac 5m 57s root in the patch failed.
          -0 checkstyle 2m 38s root: The patch generated 39 new + 2837 unchanged - 11 fixed = 2876 total (was 2848)
          +1 mvnsite 9m 55s the patch passed
          +1 mvneclipse 1m 7s the patch passed
          -1 whitespace 0m 0s The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui .
          -1 findbugs 0m 16s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          -1 javadoc 4m 23s root generated 11198 new + 0 unchanged - 0 fixed = 11198 total (was 0)
          -1 unit 140m 5s root in the patch failed.
          -1 asflicense 0m 26s The patch generated 9 ASF License warnings.
          247m 41s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.datanode.TestDataNodeMXBean
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838594/HDFS-10872.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle cc
          uname Linux e968e550d708 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 503e73e
          Default Java 1.8.0_101
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/branch-compile-root.txt
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-compile-root.txt
          cc https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/diff-javadoc-javadoc-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17530/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-tools/hadoop-openstack . U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17530/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 23 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 9m 22s trunk passed -1 compile 9m 10s root in trunk failed. +1 checkstyle 2m 31s trunk passed +1 mvnsite 11m 32s trunk passed +1 mvneclipse 4m 34s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-cloud-storage-project hadoop-cloud-storage-project/hadoop-cloud-storage -1 findbugs 0m 18s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) +1 javadoc 5m 41s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 15m 30s the patch passed -1 compile 5m 57s root in the patch failed. -1 cc 5m 57s root in the patch failed. -1 javac 5m 57s root in the patch failed. -0 checkstyle 2m 38s root: The patch generated 39 new + 2837 unchanged - 11 fixed = 2876 total (was 2848) +1 mvnsite 9m 55s the patch passed +1 mvneclipse 1m 7s the patch passed -1 whitespace 0m 0s The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui . -1 findbugs 0m 16s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) -1 javadoc 4m 23s root generated 11198 new + 0 unchanged - 0 fixed = 11198 total (was 0) -1 unit 140m 5s root in the patch failed. -1 asflicense 0m 26s The patch generated 9 ASF License warnings. 247m 41s Reason Tests Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.datanode.TestDirectoryScanner Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838594/HDFS-10872.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle cc uname Linux e968e550d708 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 503e73e Default Java 1.8.0_101 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-compile-root.txt cc https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/diff-javadoc-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17530/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17530/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-tools/hadoop-openstack . U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17530/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Thanks for looking, Mingliang Liu.

          Attaching v010 patch with updated names for the metrics: FSN(Read|Write)LockOperationName. I've also attached an example of what the outputted metrics look like on the JMX page (jmx-output).

          Show
          xkrogen Erik Krogen added a comment - Thanks for looking, Mingliang Liu . Attaching v010 patch with updated names for the metrics: FSN(Read|Write)LockOperationName . I've also attached an example of what the outputted metrics look like on the JMX page ( jmx-output ).
          Hide
          liuml07 Mingliang Liu added a comment -

          still +1 Thanks for updating the code.

          Show
          liuml07 Mingliang Liu added a comment - still +1 Thanks for updating the code.
          Hide
          zhz Zhe Zhang added a comment -

          Looks like Yetus was confused because the newest file was the JMX output. I'm attaching v11 patch which is a dup of v10.

          Show
          zhz Zhe Zhang added a comment - Looks like Yetus was confused because the newest file was the JMX output. I'm attaching v11 patch which is a dup of v10.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for taking care of this. I usually don't bump the patch version number; the JIRA will show old patch of same version (well, name) as gray so it's not confusing anyway.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for taking care of this. I usually don't bump the patch version number; the JIRA will show old patch of same version (well, name) as gray so it's not confusing anyway.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 36s Maven dependency ordering for branch
          +1 mvninstall 6m 58s trunk passed
          +1 compile 11m 45s trunk passed
          +1 checkstyle 1m 48s trunk passed
          +1 mvnsite 2m 22s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
          +1 findbugs 3m 21s trunk passed
          +1 javadoc 1m 40s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 44s the patch passed
          -1 compile 5m 5s root in the patch failed.
          -1 javac 5m 5s root in the patch failed.
          -0 checkstyle 1m 45s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862)
          +1 mvnsite 2m 14s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
          +1 findbugs 3m 41s the patch passed
          +1 javadoc 1m 35s the patch passed
          +1 unit 9m 14s hadoop-common in the patch passed.
          -1 unit 108m 43s hadoop-hdfs in the patch failed.
          -1 unit 0m 16s hadoop-yarn-ui in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          167m 8s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.TestDelegationTokenFetcher
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838646/HDFS-10872.011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux eeaf5da60b3b 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2ee18fc
          Default Java 1.8.0_101
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17541/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17541/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 36s Maven dependency ordering for branch +1 mvninstall 6m 58s trunk passed +1 compile 11m 45s trunk passed +1 checkstyle 1m 48s trunk passed +1 mvnsite 2m 22s trunk passed +1 mvneclipse 0m 38s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui +1 findbugs 3m 21s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 44s the patch passed -1 compile 5m 5s root in the patch failed. -1 javac 5m 5s root in the patch failed. -0 checkstyle 1m 45s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862) +1 mvnsite 2m 14s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui +1 findbugs 3m 41s the patch passed +1 javadoc 1m 35s the patch passed +1 unit 9m 14s hadoop-common in the patch passed. -1 unit 108m 43s hadoop-hdfs in the patch failed. -1 unit 0m 16s hadoop-yarn-ui in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 167m 8s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.datanode.TestDirectoryScanner Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838646/HDFS-10872.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux eeaf5da60b3b 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2ee18fc Default Java 1.8.0_101 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17541/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17541/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17541/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 38s Maven dependency ordering for branch
          +1 mvninstall 8m 6s trunk passed
          +1 compile 11m 3s trunk passed
          +1 checkstyle 1m 45s trunk passed
          +1 mvnsite 2m 33s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
          +1 findbugs 3m 29s trunk passed
          +1 javadoc 1m 38s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 37s the patch passed
          -1 compile 5m 17s root in the patch failed.
          -1 javac 5m 17s root in the patch failed.
          -0 checkstyle 1m 40s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862)
          +1 mvnsite 2m 11s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
          +1 findbugs 3m 47s the patch passed
          +1 javadoc 1m 43s the patch passed
          +1 unit 9m 27s hadoop-common in the patch passed.
          -1 unit 103m 58s hadoop-hdfs in the patch failed.
          -1 unit 0m 11s hadoop-yarn-ui in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          163m 15s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.TestDelegationTokenFetcher
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.namenode.ha.TestHAAppend



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838646/HDFS-10872.011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux f3b1c71a79c8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2ee18fc
          Default Java 1.8.0_101
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17542/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17542/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 38s Maven dependency ordering for branch +1 mvninstall 8m 6s trunk passed +1 compile 11m 3s trunk passed +1 checkstyle 1m 45s trunk passed +1 mvnsite 2m 33s trunk passed +1 mvneclipse 0m 39s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui +1 findbugs 3m 29s trunk passed +1 javadoc 1m 38s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 37s the patch passed -1 compile 5m 17s root in the patch failed. -1 javac 5m 17s root in the patch failed. -0 checkstyle 1m 40s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862) +1 mvnsite 2m 11s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui +1 findbugs 3m 47s the patch passed +1 javadoc 1m 43s the patch passed +1 unit 9m 27s hadoop-common in the patch passed. -1 unit 103m 58s hadoop-hdfs in the patch failed. -1 unit 0m 11s hadoop-yarn-ui in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 163m 15s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838646/HDFS-10872.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux f3b1c71a79c8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2ee18fc Default Java 1.8.0_101 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17542/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17542/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17542/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Looks like when going from v009 to v010 patch I accidentally re-included the YARN-5868 fix. Attaching v012 patch with that removed. Sorry for the mixup.

          Show
          xkrogen Erik Krogen added a comment - Looks like when going from v009 to v010 patch I accidentally re-included the YARN-5868 fix. Attaching v012 patch with that removed. Sorry for the mixup.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the tip Mingliang

          Show
          zhz Zhe Zhang added a comment - Thanks for the tip Mingliang
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 35s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 9m 30s trunk passed
          +1 checkstyle 1m 43s trunk passed
          +1 mvnsite 1m 59s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 3m 11s trunk passed
          +1 javadoc 1m 33s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 23s the patch passed
          +1 compile 9m 14s the patch passed
          +1 javac 9m 14s the patch passed
          -0 checkstyle 1m 43s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862)
          +1 mvnsite 1m 54s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 25s the patch passed
          +1 javadoc 1m 32s the patch passed
          +1 unit 8m 23s hadoop-common in the patch passed.
          -1 unit 79m 18s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          136m 50s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838788/HDFS-10872.012.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 5c6ac6a2acba 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6efb8c9
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17552/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17552/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17552/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17552/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 35s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 9m 30s trunk passed +1 checkstyle 1m 43s trunk passed +1 mvnsite 1m 59s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 3m 11s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 9m 14s the patch passed +1 javac 9m 14s the patch passed -0 checkstyle 1m 43s root: The patch generated 2 new + 862 unchanged - 0 fixed = 864 total (was 862) +1 mvnsite 1m 54s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 25s the patch passed +1 javadoc 1m 32s the patch passed +1 unit 8m 23s hadoop-common in the patch passed. -1 unit 79m 18s hadoop-hdfs in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 136m 50s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838788/HDFS-10872.012.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 5c6ac6a2acba 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6efb8c9 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17552/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17552/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17552/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17552/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik. +1 on v12 patch. I did another check of the metrics from local cluster, they look good. I just pushed to trunk, working on branch backports. Thanks for the great work!

          Show
          zhz Zhe Zhang added a comment - Thanks Erik. +1 on v12 patch. I did another check of the metrics from local cluster, they look good. I just pushed to trunk, working on branch backports. Thanks for the great work!
          Hide
          zhz Zhe Zhang added a comment -

          Actually FSNamesystem has many conflicts – though most of them look pretty mechanical. Erik Krogen Do you mind creating a branch-2 patch? I guess we should also wait for branch-2 Jenkins precommit run.

          Show
          zhz Zhe Zhang added a comment - Actually FSNamesystem has many conflicts – though most of them look pretty mechanical. Erik Krogen Do you mind creating a branch-2 patch? I guess we should also wait for branch-2 Jenkins precommit run.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10832 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10832/)
          HDFS-10872. Add MutableRate metrics for FSNamesystemLock operations. (zhz: rev ff0b99eafeda035ebe0dc82cfe689808047a8893)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemLock.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10832 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10832/ ) HDFS-10872 . Add MutableRate metrics for FSNamesystemLock operations. (zhz: rev ff0b99eafeda035ebe0dc82cfe689808047a8893) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemLock.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
          Hide
          xkrogen Erik Krogen added a comment -

          Thank you for the commit, Zhe Zhang!

          I've attached a branch-2 patch. You were correct that the conflicts were just mechanical fixes - touching so many lines in FSNamesystem in a trivial way (just adding the operation name) caused many code conflicts but no real logical conflicts.

          Show
          xkrogen Erik Krogen added a comment - Thank you for the commit, Zhe Zhang ! I've attached a branch-2 patch. You were correct that the conflicts were just mechanical fixes - touching so many lines in FSNamesystem in a trivial way (just adding the operation name) caused many code conflicts but no real logical conflicts.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 2s Docker failed to build yetus/hadoop:b59b8b7.



          Subsystem Report/Notes
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838829/HDFS-10872-branch-2.012.patch
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17556/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 2s Docker failed to build yetus/hadoop:b59b8b7. Subsystem Report/Notes JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838829/HDFS-10872-branch-2.012.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17556/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik. Local cluster on branch-2 patch looks good. I just retriggered Jenkins.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik. Local cluster on branch-2 patch looks good. I just retriggered Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 47s branch-2 passed
          +1 compile 6m 31s branch-2 passed with JDK v1.8.0_101
          +1 compile 6m 57s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 36s branch-2 passed
          +1 mvnsite 1m 54s branch-2 passed
          +1 mvneclipse 0m 29s branch-2 passed
          +1 findbugs 3m 41s branch-2 passed
          +1 javadoc 1m 51s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 2m 36s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 5m 57s the patch passed with JDK v1.8.0_101
          +1 javac 5m 57s the patch passed
          +1 compile 6m 56s the patch passed with JDK v1.7.0_111
          +1 javac 6m 56s the patch passed
          -0 checkstyle 1m 41s root: The patch generated 4 new + 859 unchanged - 0 fixed = 863 total (was 859)
          +1 mvnsite 2m 1s the patch passed
          +1 mvneclipse 0m 33s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 3s the patch passed
          +1 javadoc 1m 51s the patch passed with JDK v1.8.0_101
          +1 javadoc 2m 47s the patch passed with JDK v1.7.0_111
          +1 unit 10m 6s hadoop-common in the patch passed with JDK v1.7.0_111.
          +1 unit 67m 47s hadoop-hdfs in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          223m 50s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838829/HDFS-10872-branch-2.012.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4873aa8f19f3 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 399abe7
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17557/artifact/patchprocess/diff-checkstyle-root.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17557/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17557/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 12s Maven dependency ordering for branch +1 mvninstall 6m 47s branch-2 passed +1 compile 6m 31s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 57s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 36s branch-2 passed +1 mvnsite 1m 54s branch-2 passed +1 mvneclipse 0m 29s branch-2 passed +1 findbugs 3m 41s branch-2 passed +1 javadoc 1m 51s branch-2 passed with JDK v1.8.0_101 +1 javadoc 2m 36s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 5m 57s the patch passed with JDK v1.8.0_101 +1 javac 5m 57s the patch passed +1 compile 6m 56s the patch passed with JDK v1.7.0_111 +1 javac 6m 56s the patch passed -0 checkstyle 1m 41s root: The patch generated 4 new + 859 unchanged - 0 fixed = 863 total (was 859) +1 mvnsite 2m 1s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 3s the patch passed +1 javadoc 1m 51s the patch passed with JDK v1.8.0_101 +1 javadoc 2m 47s the patch passed with JDK v1.7.0_111 +1 unit 10m 6s hadoop-common in the patch passed with JDK v1.7.0_111. +1 unit 67m 47s hadoop-hdfs in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 223m 50s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838829/HDFS-10872-branch-2.012.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4873aa8f19f3 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 399abe7 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17557/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17557/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17557/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Looks like I introduced two small checkstyle issues when I fixed branch-2 conflicts. Attaching branch-2 v13 patch to fix.

          Show
          xkrogen Erik Krogen added a comment - Looks like I introduced two small checkstyle issues when I fixed branch-2 conflicts. Attaching branch-2 v13 patch to fix.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik! I manually verified the 2 checkstyle issues. I just committed to branch-2.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik! I manually verified the 2 checkstyle issues. I just committed to branch-2.
          Hide
          xkrogen Erik Krogen added a comment -

          Attaching patch for branch-2.7 (v13).

          Show
          xkrogen Erik Krogen added a comment - Attaching patch for branch-2.7 (v13).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 43s Maven dependency ordering for branch
          +1 mvninstall 7m 2s branch-2 passed
          +1 compile 7m 13s branch-2 passed with JDK v1.8.0_101
          +1 compile 7m 44s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 47s branch-2 passed
          +1 mvnsite 2m 3s branch-2 passed
          +1 mvneclipse 0m 35s branch-2 passed
          +1 findbugs 3m 54s branch-2 passed
          +1 javadoc 2m 0s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 3m 2s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 47s the patch passed
          +1 compile 7m 18s the patch passed with JDK v1.8.0_101
          +1 javac 7m 18s the patch passed
          +1 compile 7m 47s the patch passed with JDK v1.7.0_111
          +1 javac 7m 47s the patch passed
          -0 checkstyle 1m 55s root: The patch generated 2 new + 859 unchanged - 0 fixed = 861 total (was 859)
          +1 mvnsite 2m 8s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 30s the patch passed
          +1 javadoc 2m 3s the patch passed with JDK v1.8.0_101
          +1 javadoc 2m 59s the patch passed with JDK v1.7.0_111
          +1 unit 10m 21s hadoop-common in the patch passed with JDK v1.7.0_111.
          -1 unit 85m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          245m 47s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.TestRollingUpgrade
          JDK v1.7.0_111 Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838876/HDFS-10872-branch-2.013.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 7cb452bee571 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 5ffb26c
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17564/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17564/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17564/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17564/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 7m 2s branch-2 passed +1 compile 7m 13s branch-2 passed with JDK v1.8.0_101 +1 compile 7m 44s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 47s branch-2 passed +1 mvnsite 2m 3s branch-2 passed +1 mvneclipse 0m 35s branch-2 passed +1 findbugs 3m 54s branch-2 passed +1 javadoc 2m 0s branch-2 passed with JDK v1.8.0_101 +1 javadoc 3m 2s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 47s the patch passed +1 compile 7m 18s the patch passed with JDK v1.8.0_101 +1 javac 7m 18s the patch passed +1 compile 7m 47s the patch passed with JDK v1.7.0_111 +1 javac 7m 47s the patch passed -0 checkstyle 1m 55s root: The patch generated 2 new + 859 unchanged - 0 fixed = 861 total (was 859) +1 mvnsite 2m 8s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 30s the patch passed +1 javadoc 2m 3s the patch passed with JDK v1.8.0_101 +1 javadoc 2m 59s the patch passed with JDK v1.7.0_111 +1 unit 10m 21s hadoop-common in the patch passed with JDK v1.7.0_111. -1 unit 85m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 245m 47s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.TestRollingUpgrade JDK v1.7.0_111 Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838876/HDFS-10872-branch-2.013.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 7cb452bee571 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 5ffb26c Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17564/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17564/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17564/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17564/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 50s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 8m 14s branch-2.7 passed
          +1 compile 6m 20s branch-2.7 passed with JDK v1.8.0_111
          +1 compile 6m 47s branch-2.7 passed with JDK v1.7.0_111
          +1 checkstyle 1m 38s branch-2.7 passed
          +1 mvnsite 1m 55s branch-2.7 passed
          +1 mvneclipse 0m 33s branch-2.7 passed
          -1 findbugs 1m 49s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings.
          +1 javadoc 1m 55s branch-2.7 passed with JDK v1.8.0_111
          +1 javadoc 2m 53s branch-2.7 passed with JDK v1.7.0_111
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 43s the patch passed
          +1 compile 6m 22s the patch passed with JDK v1.8.0_111
          +1 javac 6m 22s the patch passed
          +1 compile 7m 7s the patch passed with JDK v1.7.0_111
          +1 javac 7m 7s the patch passed
          -0 checkstyle 1m 36s root: The patch generated 5 new + 1115 unchanged - 1 fixed = 1120 total (was 1116)
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          -1 whitespace 0m 0s The patch has 7079 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 3m 34s The patch 185 line(s) with tabs.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 5m 22s the patch passed
          +1 javadoc 1m 56s the patch passed with JDK v1.8.0_111
          +1 javadoc 2m 56s the patch passed with JDK v1.7.0_111
          -1 unit 21m 14s hadoop-common in the patch failed with JDK v1.7.0_111.
          -1 unit 46m 32s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          -1 asflicense 0m 21s The patch generated 3 ASF License warnings.
          221m 33s



          Reason Tests
          JDK v1.8.0_111 Failed junit tests hadoop.util.bloom.TestBloomFilters
            hadoop.io.TestBloomMapFile
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.ipc.TestCallQueueManager
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
          JDK v1.8.0_111 Timed out junit tests org.apache.hadoop.conf.TestConfiguration
          JDK v1.7.0_111 Failed junit tests hadoop.util.bloom.TestBloomFilters
            hadoop.io.TestBloomMapFile
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.io.TestMapFile
            hadoop.hdfs.web.TestWebHdfsFileSystemContract
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          JDK v1.7.0_111 Timed out junit tests org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
            org.apache.hadoop.conf.TestConfiguration



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-10872
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838888/HDFS-10872-branch-2.7.013.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ce0fe652df32 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 81bf6f2
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_111.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17566/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17566/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 50s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 8m 14s branch-2.7 passed +1 compile 6m 20s branch-2.7 passed with JDK v1.8.0_111 +1 compile 6m 47s branch-2.7 passed with JDK v1.7.0_111 +1 checkstyle 1m 38s branch-2.7 passed +1 mvnsite 1m 55s branch-2.7 passed +1 mvneclipse 0m 33s branch-2.7 passed -1 findbugs 1m 49s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings. +1 javadoc 1m 55s branch-2.7 passed with JDK v1.8.0_111 +1 javadoc 2m 53s branch-2.7 passed with JDK v1.7.0_111 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 43s the patch passed +1 compile 6m 22s the patch passed with JDK v1.8.0_111 +1 javac 6m 22s the patch passed +1 compile 7m 7s the patch passed with JDK v1.7.0_111 +1 javac 7m 7s the patch passed -0 checkstyle 1m 36s root: The patch generated 5 new + 1115 unchanged - 1 fixed = 1120 total (was 1116) +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 29s the patch passed -1 whitespace 0m 0s The patch has 7079 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 3m 34s The patch 185 line(s) with tabs. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 22s the patch passed +1 javadoc 1m 56s the patch passed with JDK v1.8.0_111 +1 javadoc 2m 56s the patch passed with JDK v1.7.0_111 -1 unit 21m 14s hadoop-common in the patch failed with JDK v1.7.0_111. -1 unit 46m 32s hadoop-hdfs in the patch failed with JDK v1.7.0_111. -1 asflicense 0m 21s The patch generated 3 ASF License warnings. 221m 33s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.util.bloom.TestBloomFilters   hadoop.io.TestBloomMapFile   hadoop.ha.TestZKFailoverControllerStress   hadoop.ipc.TestCallQueueManager   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.blockmanagement.TestPendingReplication JDK v1.8.0_111 Timed out junit tests org.apache.hadoop.conf.TestConfiguration JDK v1.7.0_111 Failed junit tests hadoop.util.bloom.TestBloomFilters   hadoop.io.TestBloomMapFile   hadoop.ha.TestZKFailoverControllerStress   hadoop.io.TestMapFile   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_111 Timed out junit tests org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken   org.apache.hadoop.conf.TestConfiguration Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-10872 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838888/HDFS-10872-branch-2.7.013.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ce0fe652df32 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 81bf6f2 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_111.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17566/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17566/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17566/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik! I just verified the reported branch-2.7 test failures (non reproducible locally), as well as local cluster, and committed to branch-2.7.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik! I just verified the reported branch-2.7 test failures (non reproducible locally), as well as local cluster, and committed to branch-2.7.

            People

            • Assignee:
              xkrogen Erik Krogen
              Reporter:
              xkrogen Erik Krogen
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development