Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8873

Allow the directoryScanner to be rate-limited

    Details

    • Target Version/s:

      Description

      The new 2-level directory layout can make directory scans expensive in terms of disk seeks (see HDFS-8791) for details.

      It would be good if the directoryScanner() had a configurable duty cycle that would reduce its impact on disk performance (much like the approach in HDFS-8617).

      Without such a throttle, disks can go 100% busy for many minutes at a time (assuming the common case of all inodes in cache but no directory blocks cached, 64K seeks are required for full directory listing which translates to 655 seconds)

      1. HDFS-8873.001.patch
        56 kB
        Daniel Templeton
      2. HDFS-8873.002.patch
        57 kB
        Daniel Templeton
      3. HDFS-8873.003.patch
        56 kB
        Daniel Templeton
      4. HDFS-8873.004.patch
        57 kB
        Daniel Templeton
      5. HDFS-8873.005.patch
        29 kB
        Daniel Templeton
      6. HDFS-8873.006.patch
        30 kB
        Daniel Templeton
      7. HDFS-8873.007.patch
        34 kB
        Daniel Templeton
      8. HDFS-8873.008.patch
        34 kB
        Daniel Templeton
      9. HDFS-8873.009.patch
        34 kB
        Daniel Templeton

        Issue Links

          Activity

          Hide
          templedf Daniel Templeton added a comment -

          Suggestion from HDFS-8989:

          We should allow rate-limiting the DirectoryScanner via time-slicing, so that it only runs every 100 ms out of 1 s (or whatever other percentage we configure.)

          Show
          templedf Daniel Templeton added a comment - Suggestion from HDFS-8989 : We should allow rate-limiting the DirectoryScanner via time-slicing, so that it only runs every 100 ms out of 1 s (or whatever other percentage we configure.)
          Hide
          templedf Daniel Templeton added a comment -

          Here's a patch to add a time-based throttle. The changes are a little large because I had to first make the scanner throttlable. I also took the liberty of fleshing out the javadocs. The throttling approach is explained in the comments and javadocs.

          Show
          templedf Daniel Templeton added a comment - Here's a patch to add a time-based throttle. The changes are a little large because I had to first make the scanner throttlable. I also took the liberty of fleshing out the javadocs. The throttling approach is explained in the comments and javadocs.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 6s There were no new javac warning messages.
          +1 javadoc 10m 25s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 22s The applied patch generated 20 new checkstyle issues (total was 442, now 441).
          -1 whitespace 0m 7s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 18s Pre-build of native portion
          -1 hdfs tests 73m 1s Tests failed in hadoop-hdfs.
              119m 6s  



          Reason Tests
          Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2
          Timed out tests org.apache.hadoop.hdfs.TestDFSFinalize



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755474/HDFS-8873.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ea4bb27
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12403/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12403/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12403/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12403/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12403/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 25s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 20 new checkstyle issues (total was 442, now 441). -1 whitespace 0m 7s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 18s Pre-build of native portion -1 hdfs tests 73m 1s Tests failed in hadoop-hdfs.     119m 6s   Reason Tests Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2 Timed out tests org.apache.hadoop.hdfs.TestDFSFinalize Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755474/HDFS-8873.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ea4bb27 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12403/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12403/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12403/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12403/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12403/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Resolved some of the checkstyle issues.

          On the line too long comments, it's in the DFSConfig file where every line is too long. I would assume I should follow the established pattern rather than introduce a new style just for checkstyle.

          On the comments about making the fields private and adding accessors, since the fields are all final, I don't see need. Anyone disagree.

          The whitespace issue wasn't something I introduced, so I don't know why it's complaining.

          Show
          templedf Daniel Templeton added a comment - Resolved some of the checkstyle issues. On the line too long comments, it's in the DFSConfig file where every line is too long. I would assume I should follow the established pattern rather than introduce a new style just for checkstyle. On the comments about making the fields private and adding accessors, since the fields are all final, I don't see need. Anyone disagree. The whitespace issue wasn't something I introduced, so I don't know why it's complaining.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Good work, Daniel Templeton.

            public static final String  DFS_DATANODE_DIRECTORYSCAN_THROTTLE_KEY = "dfs.datanode.directoryscan.throttle";
          

          Should be "throttle.type"?

          649 <property>
          650	  <name>dfs.datanode.directoryscan.throttle.limit</name>
          651	  <value>0</value>
          652	  <description>The limit setting for the report compiler threads throttle. The
          653	  meaning of this setting is determined by the
          654	  dfs.datanode.directoryscan.throttle setting. In all cases, setting this limit
          655	  to 0 disables compiler thread throttling. 0 is the default setting.
          656	  </description>
          657	</property>
          

          I would prefer to see per-type configuration keys that are more descriptive. For example, dfs.datanode.directoryscan.timeslice.throttle.ms.per.sec. If we invent more throttle types later, we can always add more configuration key names.

          testTimesliceThrottle: please copy the Configuration object and change the copy rather than mutating TestDirectoryScanner#CONF.

          604	      // Waiting should be about 4x running.
          605	      ratio = (float)scanner.timeWaiting.get() / scanner.timeRunning.get();
          606	
          607	      LOG.info("RATIO: " + ratio);
          608	      assertTrue("Throttle is too restrictive", ratio <= 4.5);
          609	      assertTrue("Throttle is too permissive", ratio >= 3.0);
          

          I'm a bit concerned that other tests running on the same machine, or GCs could cause delays here that would make the test fail. Perhaps we could do this in a loop and keep retrying until we found that the ratio was correct?

          84	  @VisibleForTesting
          85	  final AtomicLong timeRunning = new AtomicLong(0L);
          

          Should be "timeRunningMs" to reflect the fact that this interval is in milliseconds. Similar with "timeWaiting"

          115	    public static ThrottleType parse(String type) {
          116	      if (type.trim().equalsIgnoreCase(TIMESLICE.toString())) {
          117	        return TIMESLICE;
          118	      } else {
          119	        return NONE;
          120	      }
          121	    }
          

          We should log an ERROR message if we can't understand the throttle type. Silently defaulting to doing nothing is not behavior most users will appreciate.

          DirectoryScanner.java: there are some unnecessary whitespace changes.

          TimesliceThrottler: maybe TimeSliceThrottlerTask is a more appropriate name here? I like to think of executors as scheduling tasks. Arguably the throttler state is all contained outside the class so it's not really "the throttler."

          1121	      } catch (Throwable t) {
          1122	        LOG.error("Throttle thread died unexpectedly", t);
          1123	
          1124	        if (t instanceof Error) {
          1125	          throw t;
          1126	        }
          1127	      }
          

          What's the purpose of rethrowing exceptions here?

            private static class Throttle extends Semaphore {
          

          While this works, it feels more natural to use a boolean + condition variable here.

          try {
            lock.lock();
            while (blockThreads) {
              cond.wait();
            }
          } finally {
            lock.unlock();
          }
          
          74	  private final ThrottleType throttleType;
          75	  private final int throttleLimit;
          76	  private ScheduledExecutorService throttleThread;
          77	  private Semaphore runningThreads = new Semaphore(0);
          78	  private volatile Throttle throttle;
          

          It feels like this state should be encapsulated inside the Throttler itself.

          860	            // Give our parent a chance to block us for throttling
          861	            if (throttle != null) {
          862	              throttle.start();
          863	            }
          

          Can we just say that Throttler is always non-null, but sometimes we have a NoOpThrottler? I don't like all this checking for null business.

          You could get rid of the type enum and all those explicit type checks, and just have a factory function inside an interface that creates the appropriate kind of Throttler object from the Configuration (no-op, timeslice, etc).

          Show
          cmccabe Colin P. McCabe added a comment - Good work, Daniel Templeton . public static final String DFS_DATANODE_DIRECTORYSCAN_THROTTLE_KEY = "dfs.datanode.directoryscan.throttle" ; Should be "throttle.type"? 649 <property> 650 <name>dfs.datanode.directoryscan.throttle.limit</name> 651 <value>0</value> 652 <description>The limit setting for the report compiler threads throttle. The 653 meaning of this setting is determined by the 654 dfs.datanode.directoryscan.throttle setting. In all cases, setting this limit 655 to 0 disables compiler thread throttling. 0 is the default setting. 656 </description> 657 </property> I would prefer to see per-type configuration keys that are more descriptive. For example, dfs.datanode.directoryscan.timeslice.throttle.ms.per.sec . If we invent more throttle types later, we can always add more configuration key names. testTimesliceThrottle: please copy the Configuration object and change the copy rather than mutating TestDirectoryScanner#CONF . 604 // Waiting should be about 4x running. 605 ratio = ( float )scanner.timeWaiting.get() / scanner.timeRunning.get(); 606 607 LOG.info( "RATIO: " + ratio); 608 assertTrue( "Throttle is too restrictive" , ratio <= 4.5); 609 assertTrue( "Throttle is too permissive" , ratio >= 3.0); I'm a bit concerned that other tests running on the same machine, or GCs could cause delays here that would make the test fail. Perhaps we could do this in a loop and keep retrying until we found that the ratio was correct? 84 @VisibleForTesting 85 final AtomicLong timeRunning = new AtomicLong(0L); Should be "timeRunningMs" to reflect the fact that this interval is in milliseconds. Similar with "timeWaiting" 115 public static ThrottleType parse( String type) { 116 if (type.trim().equalsIgnoreCase(TIMESLICE.toString())) { 117 return TIMESLICE; 118 } else { 119 return NONE; 120 } 121 } We should log an ERROR message if we can't understand the throttle type. Silently defaulting to doing nothing is not behavior most users will appreciate. DirectoryScanner.java: there are some unnecessary whitespace changes. TimesliceThrottler: maybe TimeSliceThrottlerTask is a more appropriate name here? I like to think of executors as scheduling tasks. Arguably the throttler state is all contained outside the class so it's not really "the throttler." 1121 } catch (Throwable t) { 1122 LOG.error( "Throttle thread died unexpectedly" , t); 1123 1124 if (t instanceof Error) { 1125 throw t; 1126 } 1127 } What's the purpose of rethrowing exceptions here? private static class Throttle extends Semaphore { While this works, it feels more natural to use a boolean + condition variable here. try { lock.lock(); while (blockThreads) { cond.wait(); } } finally { lock.unlock(); } 74 private final ThrottleType throttleType; 75 private final int throttleLimit; 76 private ScheduledExecutorService throttleThread; 77 private Semaphore runningThreads = new Semaphore(0); 78 private volatile Throttle throttle; It feels like this state should be encapsulated inside the Throttler itself. 860 // Give our parent a chance to block us for throttling 861 if (throttle != null ) { 862 throttle.start(); 863 } Can we just say that Throttler is always non-null, but sometimes we have a NoOpThrottler ? I don't like all this checking for null business. You could get rid of the type enum and all those explicit type checks, and just have a factory function inside an interface that creates the appropriate kind of Throttler object from the Configuration (no-op, timeslice, etc).
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 56s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 1s There were no new javac warning messages.
          +1 javadoc 10m 1s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 34s There were no new checkstyle issues.
          -1 whitespace 0m 7s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 191m 53s Tests failed in hadoop-hdfs.
              234m 46s  



          Reason Tests
          Failed unit tests hadoop.tools.TestJMXGet
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755501/HDFS-8873.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 9538af0
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12406/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12406/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12406/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12406/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 56s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 1s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 34s There were no new checkstyle issues. -1 whitespace 0m 7s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 191m 53s Tests failed in hadoop-hdfs.     234m 46s   Reason Tests Failed unit tests hadoop.tools.TestJMXGet   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755501/HDFS-8873.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9538af0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12406/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12406/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12406/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12406/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Colin P. McCabe, I implemented all of your suggestions except the throttle limit property name.

          I would prefer to see per-type configuration keys that are more descriptive. For example, dfs.datanode.directoryscan.timeslice.throttle.ms.per.sec.

          Encapsulating the throttle stuff requires a reasonable abstraction of what a throttle is. The various kinds of throttles (time, file, iops, ...) are all pretty different and aren't easy to overlay with a single abstraction. I decided to give up on the idea of making the throttle type selectable. The limit therefore always means the same thing, and so I think it's fair to leave it's name as is.

          What's the purpose of rethrowing exceptions here?

          I'm only rethrowing Errors, which is the right thing to do. It would be bad if a thread caught an OOME and discarded it.

          Show
          templedf Daniel Templeton added a comment - Colin P. McCabe , I implemented all of your suggestions except the throttle limit property name. I would prefer to see per-type configuration keys that are more descriptive. For example, dfs.datanode.directoryscan.timeslice.throttle.ms.per.sec. Encapsulating the throttle stuff requires a reasonable abstraction of what a throttle is. The various kinds of throttles (time, file, iops, ...) are all pretty different and aren't easy to overlay with a single abstraction. I decided to give up on the idea of making the throttle type selectable. The limit therefore always means the same thing, and so I think it's fair to leave it's name as is. What's the purpose of rethrowing exceptions here? I'm only rethrowing Errors, which is the right thing to do. It would be bad if a thread caught an OOME and discarded it.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 44s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 55s There were no new javac warning messages.
          +1 javadoc 10m 7s There were no new javadoc warning messages.
          -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 21s The applied patch generated 7 new checkstyle issues (total was 442, now 425).
          -1 whitespace 0m 6s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 2m 34s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 171m 51s Tests failed in hadoop-hdfs.
              217m 14s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.tools.TestJMXGet
          Timed out tests org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755763/HDFS-8873.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6955771
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12428/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12428/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 44s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 21s The applied patch generated 7 new checkstyle issues (total was 442, now 425). -1 whitespace 0m 6s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 2m 34s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 171m 51s Tests failed in hadoop-hdfs.     217m 14s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.tools.TestJMXGet Timed out tests org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755763/HDFS-8873.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6955771 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12428/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12428/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12428/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Cleaned up a couple checkstyle and whitespace issues

          Show
          templedf Daniel Templeton added a comment - Cleaned up a couple checkstyle and whitespace issues
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 47s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 51s There were no new javac warning messages.
          +1 javadoc 10m 8s There were no new javadoc warning messages.
          -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 26s The applied patch generated 4 new checkstyle issues (total was 442, now 422).
          -1 whitespace 0m 10s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          -1 findbugs 2m 41s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 20s Pre-build of native portion
          -1 hdfs tests 168m 7s Tests failed in hadoop-hdfs.
              214m 2s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
            hadoop.hdfs.web.TestWebHDFSOAuth2
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.server.namenode.TestSaveNamespace
            hadoop.hdfs.server.datanode.TestIncrementalBrVariations
            hadoop.tools.TestJMXGet
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755813/HDFS-8873.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6955771
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12432/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12432/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 47s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 10m 8s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 26s The applied patch generated 4 new checkstyle issues (total was 442, now 422). -1 whitespace 0m 10s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 2m 41s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 20s Pre-build of native portion -1 hdfs tests 168m 7s Tests failed in hadoop-hdfs.     214m 2s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.hdfs.web.TestWebHDFSOAuth2   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.server.namenode.TestSaveNamespace   hadoop.hdfs.server.datanode.TestIncrementalBrVariations   hadoop.tools.TestJMXGet   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755813/HDFS-8873.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6955771 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12432/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12432/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12432/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Encapsulating the throttle stuff requires a reasonable abstraction of what a throttle is. The various kinds of throttles (time, file, iops, ...) are all pretty different and aren't easy to overlay with a single abstraction. I decided to give up on the idea of making the throttle type selectable. The limit therefore always means the same thing, and so I think it's fair to leave it's name as is.

          I agree that it is probably premature to have a single Throttle base class that can do all the various possible things you might want a Throttle to do. But it doesn't follow from that that we need to give up on the idea of making the throttle type selectable in the future.

          Also, configuration keys which are specified in terms of milliseconds should always end in ms. It causes a huge amount of confusion when times come without units. It is obvious to you-- the author of the code-- that it is in milliseconds, but it is not obvious to users or other developers.

          Why not just have a single class called TimeBasedThrottle which does whatever you want your time-based throttle to do? You can make it a standalone class that doesn't extend or implement anything, and even create an instance of it that does nothing when throttling is turned off. But keep the flexible configuration mechanism that we discussed earlier. That way, if someone wants to do something more elaborate later, they can.

          hadoop-hdfs-project/hadoop-hdfs/now: did you intend to put this in the patch?

          397  public static final String  DFS_DATANODE_DIRECTORYSCAN_THROTTLE_LIMIT_KEY =
          398	      "dfs.datanode.directoryscan.throttle.limit";
          

          Should end with "ms" to indicate the limit is in milliseconds. Or ideally "ms.per.sec"

          -    if (!retainDiffs) clear();
          +
          +    if (!retainDiffs) {
          +      clear();
          +    }
          

          Can we move small whitespace cleanups like this to a follow-on change? It just makes backports a pain because it creates unnecessary conflicts, and tends to obscure what this JIRA is about when people are reading the change log.

          +        } //end for
          +      } //end synchronized
          +    } // end if
          

          If we're changing this, then let's get rid of the COBOLisms. A close brace is enough to know that the block is closed.

          Show
          cmccabe Colin P. McCabe added a comment - Encapsulating the throttle stuff requires a reasonable abstraction of what a throttle is. The various kinds of throttles (time, file, iops, ...) are all pretty different and aren't easy to overlay with a single abstraction. I decided to give up on the idea of making the throttle type selectable. The limit therefore always means the same thing, and so I think it's fair to leave it's name as is. I agree that it is probably premature to have a single Throttle base class that can do all the various possible things you might want a Throttle to do. But it doesn't follow from that that we need to give up on the idea of making the throttle type selectable in the future. Also, configuration keys which are specified in terms of milliseconds should always end in ms. It causes a huge amount of confusion when times come without units. It is obvious to you-- the author of the code-- that it is in milliseconds, but it is not obvious to users or other developers. Why not just have a single class called TimeBasedThrottle which does whatever you want your time-based throttle to do? You can make it a standalone class that doesn't extend or implement anything, and even create an instance of it that does nothing when throttling is turned off. But keep the flexible configuration mechanism that we discussed earlier. That way, if someone wants to do something more elaborate later, they can. hadoop-hdfs-project/hadoop-hdfs/now : did you intend to put this in the patch? 397 public static final String DFS_DATANODE_DIRECTORYSCAN_THROTTLE_LIMIT_KEY = 398 "dfs.datanode.directoryscan.throttle.limit" ; Should end with "ms" to indicate the limit is in milliseconds. Or ideally "ms.per.sec" - if (!retainDiffs) clear(); + + if (!retainDiffs) { + clear(); + } Can we move small whitespace cleanups like this to a follow-on change? It just makes backports a pain because it creates unnecessary conflicts, and tends to obscure what this JIRA is about when people are reading the change log. + } //end for + } //end synchronized + } // end if If we're changing this, then let's get rid of the COBOLisms. A close brace is enough to know that the block is closed.
          Hide
          templedf Daniel Templeton added a comment -

          New patch. Colin P. McCabe, I addressed your concerns about key name, whitespace, and comments. I haven't done anything in the patch that will stand in the way of someone extending the throttle concept to be more flexible later, but I don't think it's worth the effort to proactively invest in doing more to enable that extension since it may well never happen.

          Show
          templedf Daniel Templeton added a comment - New patch. Colin P. McCabe , I addressed your concerns about key name, whitespace, and comments. I haven't done anything in the patch that will stand in the way of someone extending the throttle concept to be more flexible later, but I don't think it's worth the effort to proactively invest in doing more to enable that extension since it may well never happen.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 49s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 2m 3s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12756379/HDFS-8873.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0832b38
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12499/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 49s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 2m 3s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756379/HDFS-8873.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0832b38 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12499/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Reposting patch.

          Show
          templedf Daniel Templeton added a comment - Reposting patch.
          Hide
          kihwal Kihwal Lee added a comment -

          Shall we target 2.7.2?

          Show
          kihwal Kihwal Lee added a comment - Shall we target 2.7.2?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 13s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 53s There were no new javac warning messages.
          +1 javadoc 10m 14s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 34s There were no new checkstyle issues.
          -1 whitespace 0m 7s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 2m 34s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 130m 1s Tests failed in hadoop-hdfs.
              173m 34s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestParallelShortCircuitRead
            hadoop.hdfs.server.namenode.TestAllowFormat
            hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
            hadoop.hdfs.TestFileLengthOnClusterRestart
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
            hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
            hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.hdfs.server.namenode.TestCheckpoint
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestRemoteBlockReader2
            hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
            hadoop.hdfs.TestDFSStorageStateRecovery
            hadoop.hdfs.server.namenode.TestFSImageWithXAttr
            hadoop.hdfs.TestRemoteBlockReader
            hadoop.hdfs.TestMultiThreadedHflush
            hadoop.hdfs.TestBlockReaderLocal
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
            hadoop.hdfs.server.namenode.TestFSImageWithAcl
            hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
            hadoop.hdfs.TestPread
            hadoop.hdfs.server.namenode.TestFSEditLogLoader
            hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
            hadoop.hdfs.TestDFSAddressConfig
            hadoop.hdfs.server.namenode.TestFSDirectory
            hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
            hadoop.hdfs.TestParallelRead
            hadoop.hdfs.TestRestartDFS
            hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
            hadoop.hdfs.server.namenode.TestDeadDatanode
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
            hadoop.hdfs.TestFetchImage
            hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
            hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.hdfs.server.namenode.ha.TestHAMetrics
            hadoop.hdfs.TestQuota
            hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot
            hadoop.hdfs.server.namenode.TestQuotaByStorageType
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
            hadoop.hdfs.server.namenode.TestStorageRestore
            hadoop.hdfs.server.namenode.TestSaveNamespace
            hadoop.hdfs.server.namenode.TestParallelImageWrite
            hadoop.hdfs.tools.TestDebugAdmin
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.TestNameEditsConfigs
            hadoop.hdfs.server.namenode.TestNameNodeRpcServer
            hadoop.hdfs.server.namenode.TestSnapshotPathINodes
            hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
            hadoop.hdfs.TestModTime
            hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.TestDFSStartupVersions
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy
            hadoop.hdfs.TestFileAppend4
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.server.namenode.TestINodeFile
            hadoop.hdfs.TestFileAppend2
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.hdfs.server.namenode.TestAuditLogs
            hadoop.hdfs.TestInjectionForSimulatedStorage
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
            hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.namenode.TestNameNodeAcl
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
            hadoop.hdfs.TestConnCache
            hadoop.hdfs.TestGetFileChecksum
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestReservedRawPaths
            hadoop.hdfs.TestDisableConnCache
            hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.TestParallelShortCircuitReadUnCached
            hadoop.hdfs.server.namenode.TestNameNodeMXBean
            hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename
            hadoop.hdfs.server.namenode.TestAuditLogger
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.namenode.TestMalformedURLs
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestSafeMode
            hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
            hadoop.hdfs.protocolPB.TestPBHelper
            hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.namenode.TestFileContextAcl
            hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
            hadoop.hdfs.server.namenode.TestAclConfigFlag
            hadoop.hdfs.TestWriteRead
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
            hadoop.hdfs.server.namenode.TestMetaSave
            hadoop.hdfs.TestFileCreation
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
            hadoop.hdfs.TestClientReportBadBlock
            hadoop.hdfs.server.namenode.TestAddBlockRetry
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.namenode.TestBlockUnderConstruction
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.server.namenode.TestHDFSConcat
            hadoop.hdfs.TestClose
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
            hadoop.hdfs.TestDFSInputStream
            hadoop.hdfs.TestLeaseRecovery
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestWriteBlockGetsBlockLengthHint
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.hdfs.server.namenode.TestFileLimit
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
            hadoop.hdfs.TestHDFSTrash
            hadoop.hdfs.TestSmallBlock
            hadoop.hdfs.TestReplication
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
            hadoop.hdfs.server.namenode.TestEditLogRace
            hadoop.hdfs.TestFileAppend3
            hadoop.hdfs.server.namenode.TestFileContextXAttr
            hadoop.hdfs.TestPipelines
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.ha.TestHASafeMode
            hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
            hadoop.hdfs.server.namenode.TestINodeAttributeProvider
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
            hadoop.hdfs.TestBlockReaderFactory
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication
            hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestFSOutputSummer
            hadoop.hdfs.TestBlockReaderLocalLegacy
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
            hadoop.hdfs.server.namenode.TestXAttrConfigFlag
            hadoop.hdfs.TestFileConcurrentReader
            hadoop.hdfs.TestAppendDifferentChecksum
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.namenode.TestCreateEditsLog
            hadoop.hdfs.server.namenode.TestFSNamesystemMBean
            hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
            hadoop.hdfs.server.namenode.TestNameNodeXAttr
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.TestParallelUnixDomainRead
            hadoop.hdfs.tools.TestStoragePolicyCommands
            hadoop.hdfs.TestDFSRemove
            hadoop.hdfs.server.namenode.TestHostsFiles
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.namenode.TestDeleteRace
            hadoop.hdfs.TestLease
            hadoop.hdfs.TestLargeBlock
            hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean
            hadoop.hdfs.TestBlockMissingException
          Timed out tests org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12757080/HDFS-8873.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6c6e734
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12506/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12506/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12506/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12506/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12506/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 13s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 53s There were no new javac warning messages. +1 javadoc 10m 14s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 34s There were no new checkstyle issues. -1 whitespace 0m 7s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 2m 34s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 130m 1s Tests failed in hadoop-hdfs.     173m 34s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestParallelShortCircuitRead   hadoop.hdfs.server.namenode.TestAllowFormat   hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing   hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestRemoteBlockReader2   hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.hdfs.server.namenode.TestFSImageWithXAttr   hadoop.hdfs.TestRemoteBlockReader   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.server.namenode.TestFSImageWithAcl   hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   hadoop.hdfs.TestPread   hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.hdfs.server.namenode.ha.TestQuotasWithHA   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl   hadoop.hdfs.TestDFSAddressConfig   hadoop.hdfs.server.namenode.TestFSDirectory   hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots   hadoop.hdfs.TestParallelRead   hadoop.hdfs.TestRestartDFS   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled   hadoop.hdfs.TestFetchImage   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.server.namenode.ha.TestHAMetrics   hadoop.hdfs.TestQuota   hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot   hadoop.hdfs.server.namenode.TestQuotaByStorageType   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.server.namenode.TestStorageRestore   hadoop.hdfs.server.namenode.TestSaveNamespace   hadoop.hdfs.server.namenode.TestParallelImageWrite   hadoop.hdfs.tools.TestDebugAdmin   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestNameEditsConfigs   hadoop.hdfs.server.namenode.TestNameNodeRpcServer   hadoop.hdfs.server.namenode.TestSnapshotPathINodes   hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters   hadoop.hdfs.TestModTime   hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.TestDFSStartupVersions   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy   hadoop.hdfs.TestFileAppend4   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.TestINodeFile   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.namenode.TestAuditLogs   hadoop.hdfs.TestInjectionForSimulatedStorage   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks   hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.TestNameNodeAcl   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr   hadoop.hdfs.TestConnCache   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestReservedRawPaths   hadoop.hdfs.TestDisableConnCache   hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot   hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename   hadoop.hdfs.server.namenode.TestAuditLogger   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.namenode.TestMalformedURLs   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestSafeMode   hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot   hadoop.hdfs.protocolPB.TestPBHelper   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.namenode.TestFileContextAcl   hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement   hadoop.hdfs.server.namenode.TestAclConfigFlag   hadoop.hdfs.TestWriteRead   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.hdfs.TestFileCreation   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot   hadoop.hdfs.TestClientReportBadBlock   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.namenode.TestBlockUnderConstruction   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.namenode.TestHDFSConcat   hadoop.hdfs.TestClose   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.TestDFSInputStream   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestWriteBlockGetsBlockLengthHint   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.server.namenode.TestFileLimit   hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.TestSmallBlock   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestNameNodeResourceChecker   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.server.namenode.TestEditLogRace   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.server.namenode.TestFileContextXAttr   hadoop.hdfs.TestPipelines   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot   hadoop.hdfs.server.namenode.TestINodeAttributeProvider   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps   hadoop.hdfs.TestBlockReaderFactory   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestFSOutputSummer   hadoop.hdfs.TestBlockReaderLocalLegacy   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.server.namenode.ha.TestHAStateTransitions   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots   hadoop.hdfs.server.namenode.TestXAttrConfigFlag   hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.namenode.TestCreateEditsLog   hadoop.hdfs.server.namenode.TestFSNamesystemMBean   hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary   hadoop.hdfs.server.namenode.TestNameNodeXAttr   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestParallelUnixDomainRead   hadoop.hdfs.tools.TestStoragePolicyCommands   hadoop.hdfs.TestDFSRemove   hadoop.hdfs.server.namenode.TestHostsFiles   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.TestLease   hadoop.hdfs.TestLargeBlock   hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean   hadoop.hdfs.TestBlockMissingException Timed out tests org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12757080/HDFS-8873.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6c6e734 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12506/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12506/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12506/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12506/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12506/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          [sigh] Those tests pass for me locally, so can't say why they failed.

          The whitespace error is interesting. I changed line n in the patch. Jenkins complained about the whitespace on line n+1. I fixed the whitespace on line n+1 in the next patch. Jenkins is now complaining about the whitespace on line n+2. There is no issue on line n+3, so I could correct n+2 and be done, but at that point I've made whitespace changes on two lines that I didn't otherwise touch. What's the accepted why to do it? Fix the whitespace or ignore the error?

          Show
          templedf Daniel Templeton added a comment - [sigh] Those tests pass for me locally, so can't say why they failed. The whitespace error is interesting. I changed line n in the patch. Jenkins complained about the whitespace on line n+1. I fixed the whitespace on line n+1 in the next patch. Jenkins is now complaining about the whitespace on line n+2. There is no issue on line n+3, so I could correct n+2 and be done, but at that point I've made whitespace changes on two lines that I didn't otherwise touch. What's the accepted why to do it? Fix the whitespace or ignore the error?
          Hide
          cmccabe Colin P. McCabe added a comment -

          The jenkins errors look like:

          java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.protocol.DatanodeInfo.<init>(Lorg/apache/hadoop/hdfs/protocol/DatanodeID;Ljava/lang/String;JJJJJJJJILorg/apache/hadoop/hdfs/protocol/DatanodeInfo$AdminStates;)V
          	at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:591)
          

          We've seen this before and never managed to track it down. It seems to be a bug in our Jenkins integration, possibly related to having multiple maven invocations going on at once sharing the same .m2 directory. I will re-trigger the build.

          The whitespace error is interesting. I changed line n in the patch. Jenkins complained about the whitespace on line n+1. I fixed the whitespace on line n+1 in the next patch. Jenkins is now complaining about the whitespace on line n+2

          I would say just leave it alone. If you didn't introduce the whitespace issue then don't worry about it. We really should turn off most of those checkstyle things since it provides no value.

          Show
          cmccabe Colin P. McCabe added a comment - The jenkins errors look like: java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.protocol.DatanodeInfo.<init>(Lorg/apache/hadoop/hdfs/protocol/DatanodeID;Ljava/lang/ String ;JJJJJJJJILorg/apache/hadoop/hdfs/protocol/DatanodeInfo$AdminStates;)V at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:591) We've seen this before and never managed to track it down. It seems to be a bug in our Jenkins integration, possibly related to having multiple maven invocations going on at once sharing the same .m2 directory. I will re-trigger the build. The whitespace error is interesting. I changed line n in the patch. Jenkins complained about the whitespace on line n+1. I fixed the whitespace on line n+1 in the next patch. Jenkins is now complaining about the whitespace on line n+2 I would say just leave it alone. If you didn't introduce the whitespace issue then don't worry about it. We really should turn off most of those checkstyle things since it provides no value.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 29s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 49s There were no new javac warning messages.
          +1 javadoc 9m 54s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 20s The applied patch generated 6 new checkstyle issues (total was 443, now 428).
          -1 whitespace 0m 6s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 29s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 10s Pre-build of native portion
          +1 hdfs tests 192m 33s Tests passed in hadoop-hdfs.
              237m 22s  



          Reason Tests
          FindBugs module:hadoop-hdfs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12757080/HDFS-8873.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3f82f58
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12518/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12518/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12518/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12518/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12518/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12518/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 49s There were no new javac warning messages. +1 javadoc 9m 54s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 20s The applied patch generated 6 new checkstyle issues (total was 443, now 428). -1 whitespace 0m 6s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 29s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 10s Pre-build of native portion +1 hdfs tests 192m 33s Tests passed in hadoop-hdfs.     237m 22s   Reason Tests FindBugs module:hadoop-hdfs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12757080/HDFS-8873.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3f82f58 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12518/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12518/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12518/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12518/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12518/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12518/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Daniel Templeton. Looking better.

          The configuration looks good. We had a quick offline discussion where Daniel Templeton pointed out that we could add a selectable throttle type in the future in a compatible way. So we don't need to have the throttle type selector in this patch.

          46	  public DirectoryScannerThrottle(int limit) {
          47	    if ((limit <= 0) || (limit >= 1000)) {
          

          We use "1000" a lot in this code. Can we have a constant like MS_PER_SEC to make its role clearer?

          109	  public synchronized void cycle() throws InterruptedException {
          110	    while (!open) {
          111	      try {
          112	        wait();
          113	      } catch (InterruptedException ex) {
          114	        // Ignore
          115	      }
          116	    }
          

          Why are we declaring that the function throws InterruptedException if we just catch it and do nothing?

          In ReportCompiler#run, we should probably at least call Thread.currentThread.interrupt rather than swallowing the InterruptedException completely. As described in http://www.ibm.com/developerworks/library/j-jtp05236/ , "even noncancelable tasks should attempt to preserve the interrupted status in case code higher up on the call stack wants to act on the interruption after the noncancelable task completes."

          84	  @VisibleForTesting
          85	  final Map<String, Stats> stats = new HashMap<>();
          

          It's not clear from looking at this what the String key is all about. Either document that it is a block pool id in the comment, or rename the variable to statsPerBpId or similar.

          508	          int d = 0; // index for blockpoolReport
          509	          int m = 0; // index for memReprot
          

          Rename d to blockReportIdx, m to memReportIdx? I don't normally complain about short variable names, but this loop is somewhat complex.

          683	    // No need to set an initial capacity because the number of entries will
          684	    // never be that large
          

          All very true, but you could have just initialized it with reports.size() and saved some typing

          ThrottleTask / UnthrottleTask: is there any way we could reuse these objects and avoid generating all that garbage? It seems like they are stateless (although they alter state inside the DirectoryScannerThrottle). We could probably just keep one of each inside the parent class and avoid allocating.

                long mark = Time.monotonicNow();
          

          Please make this markMs. Same for newMark.

          913	          // Skip all the files that cycle with block name until
          914	          // getting to the metafile for the block
          915	          while ((i + 1 < files.length) && files[i + 1].isFile()
          916	                  && files[i + 1].getName().startsWith(blockFile.getName())) {
          917	            i++;
          918	
          919	            if (isBlockMetaFile(blockFile.getName(), files[i].getName())) {
          920	              metaFile = files[i];
          921	              break;
          922	            }
          923	          }
          

          I realize you didn't change this in your patch, but this is a potentially very expensive operation depending on the directory size. Perhaps we should sort the array of children at the lowest level, so that we know that the meta file is the next entry, if it exists. We might want to do that in a follow-on jira.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Daniel Templeton . Looking better. The configuration looks good. We had a quick offline discussion where Daniel Templeton pointed out that we could add a selectable throttle type in the future in a compatible way. So we don't need to have the throttle type selector in this patch. 46 public DirectoryScannerThrottle( int limit) { 47 if ((limit <= 0) || (limit >= 1000)) { We use "1000" a lot in this code. Can we have a constant like MS_PER_SEC to make its role clearer? 109 public synchronized void cycle() throws InterruptedException { 110 while (!open) { 111 try { 112 wait(); 113 } catch (InterruptedException ex) { 114 // Ignore 115 } 116 } Why are we declaring that the function throws InterruptedException if we just catch it and do nothing? In ReportCompiler#run , we should probably at least call Thread.currentThread.interrupt rather than swallowing the InterruptedException completely. As described in http://www.ibm.com/developerworks/library/j-jtp05236/ , "even noncancelable tasks should attempt to preserve the interrupted status in case code higher up on the call stack wants to act on the interruption after the noncancelable task completes." 84 @VisibleForTesting 85 final Map< String , Stats> stats = new HashMap<>(); It's not clear from looking at this what the String key is all about. Either document that it is a block pool id in the comment, or rename the variable to statsPerBpId or similar. 508 int d = 0; // index for blockpoolReport 509 int m = 0; // index for memReprot Rename d to blockReportIdx, m to memReportIdx? I don't normally complain about short variable names, but this loop is somewhat complex. 683 // No need to set an initial capacity because the number of entries will 684 // never be that large All very true, but you could have just initialized it with reports.size() and saved some typing ThrottleTask / UnthrottleTask: is there any way we could reuse these objects and avoid generating all that garbage? It seems like they are stateless (although they alter state inside the DirectoryScannerThrottle). We could probably just keep one of each inside the parent class and avoid allocating. long mark = Time.monotonicNow(); Please make this markMs. Same for newMark. 913 // Skip all the files that cycle with block name until 914 // getting to the metafile for the block 915 while ((i + 1 < files.length) && files[i + 1].isFile() 916 && files[i + 1].getName().startsWith(blockFile.getName())) { 917 i++; 918 919 if (isBlockMetaFile(blockFile.getName(), files[i].getName())) { 920 metaFile = files[i]; 921 break ; 922 } 923 } I realize you didn't change this in your patch, but this is a potentially very expensive operation depending on the directory size. Perhaps we should sort the array of children at the lowest level, so that we know that the meta file is the next entry, if it exists. We might want to do that in a follow-on jira.
          Hide
          hitliuyi Yi Liu added a comment -

          We've seen this before and never managed to track it down. It seems to be a bug in our Jenkins integration, possibly related to having multiple maven invocations going on at once sharing the same .m2 directory. I will re-trigger the build.

          Yes, I think so, I see similar issue several times. If I have a HDFS patch, and also add new class in Hadoop common, then it easily fails because other builds may overwrite the hadoop common jar of my build installed. Unless we use different virtual machine, maybe through docker container to solve this problem? Maybe need support from Infra?

          Show
          hitliuyi Yi Liu added a comment - We've seen this before and never managed to track it down. It seems to be a bug in our Jenkins integration, possibly related to having multiple maven invocations going on at once sharing the same .m2 directory. I will re-trigger the build. Yes, I think so, I see similar issue several times. If I have a HDFS patch, and also add new class in Hadoop common, then it easily fails because other builds may overwrite the hadoop common jar of my build installed. Unless we use different virtual machine, maybe through docker container to solve this problem? Maybe need support from Infra?
          Hide
          templedf Daniel Templeton added a comment -

          Can we have a constant like MS_PER_SEC to make its role clearer?

          For you, Colin P. McCabe, anything.

          Why are we declaring that the function throws InterruptedException if we just catch it and do nothing?

          Good catch. (No pun intended.)

          In ReportCompiler#run, we should probably at least call Thread.currentThread.interrupt rather than swallowing the InterruptedException completely.

          OK.

          Question about the rest of your comments: Except for the last one, the rest of your comments apply to code that I touched only by indenting it. I didn't modify any of that code that I didn't have to, in the name of simplifying review by diff. Sounds like the going rule is, "you touch it, you own it," even if it's just a whitespace change. Am I reading that correctly?

          Show
          templedf Daniel Templeton added a comment - Can we have a constant like MS_PER_SEC to make its role clearer? For you, Colin P. McCabe , anything. Why are we declaring that the function throws InterruptedException if we just catch it and do nothing? Good catch. (No pun intended.) In ReportCompiler#run, we should probably at least call Thread.currentThread.interrupt rather than swallowing the InterruptedException completely. OK. Question about the rest of your comments: Except for the last one, the rest of your comments apply to code that I touched only by indenting it. I didn't modify any of that code that I didn't have to, in the name of simplifying review by diff. Sounds like the going rule is, "you touch it, you own it," even if it's just a whitespace change. Am I reading that correctly?
          Hide
          templedf Daniel Templeton added a comment -

          Oops, forgot to address the ThrottleTask comment:

          ThrottleTask / UnthrottleTask: is there any way we could reuse these objects and avoid generating all that garbage?

          Sure.

          Show
          templedf Daniel Templeton added a comment - Oops, forgot to address the ThrottleTask comment: ThrottleTask / UnthrottleTask: is there any way we could reuse these objects and avoid generating all that garbage? Sure.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Question about the rest of your comments: Except for the last one, the rest of your comments apply to code that I touched only by indenting it. I didn't modify any of that code that I didn't have to, in the name of simplifying review by diff. Sounds like the going rule is, "you touch it, you own it," even if it's just a whitespace change. Am I reading that correctly?

          That's a fair point. If you want to leave the d and m variables as-is to simplify the diff and backporting, that's fine. We should fix them at some point, but it doesn't have to be right now.

          Show
          cmccabe Colin P. McCabe added a comment - Question about the rest of your comments: Except for the last one, the rest of your comments apply to code that I touched only by indenting it. I didn't modify any of that code that I didn't have to, in the name of simplifying review by diff. Sounds like the going rule is, "you touch it, you own it," even if it's just a whitespace change. Am I reading that correctly? That's a fair point. If you want to leave the d and m variables as-is to simplify the diff and backporting, that's fine. We should fix them at some point, but it doesn't have to be right now.
          Hide
          templedf Daniel Templeton added a comment -

          Cool. I'm already planning a follow-up patch to clean up the code. I'll add that to the list.

          Show
          templedf Daniel Templeton added a comment - Cool. I'm already planning a follow-up patch to clean up the code. I'll add that to the list.
          Hide
          templedf Daniel Templeton added a comment -

          Once more with feeling. When this JIRA is resolved, I'll open another to address the style issues in the directory scanner in general.

          Show
          templedf Daniel Templeton added a comment - Once more with feeling. When this JIRA is resolved, I'll open another to address the style issues in the directory scanner in general.
          Hide
          nroberts Nathan Roberts added a comment -

          Thanks Daniel Templeton for the patch. A few comments.

          hdfs-default.xml

          • I personally would prefer the default to be 1000. In my mind 0 is a special out-of-range condition that we're allowing to mean "full rate". Just reading the default of 0 and then the first sentence of the description could easily lead one to believe the report threads are effectively off by default.
            Test
          • Any way to avoid the sleep(5000) in the test? Our tests already take a really long time, so anytime we can avoid sleeping it's better. Maybe wait at most 5 seconds for timeWaitingMs.get() to become > 0
            directoryScannerThrottle
          • Shouldn't stop call resume() instead of just notifyAll(). Will cycle() get out if we try to shutdown while in that wait?
          • Did we hit this problem with too big of hammer? Couldn't cycle() be implemented with a simple sleep? For example, with a 75% duty cycle,
            n = Time.monotonicNow() % 1000;
            if (n > 1000 * 0.75) sleep(1000- n)
            

            Seems like it could be as simple as a config and a couple of lines of code. Maybe I'm missing something or there are grander plans for the throttle.

          Show
          nroberts Nathan Roberts added a comment - Thanks Daniel Templeton for the patch. A few comments. hdfs-default.xml I personally would prefer the default to be 1000. In my mind 0 is a special out-of-range condition that we're allowing to mean "full rate". Just reading the default of 0 and then the first sentence of the description could easily lead one to believe the report threads are effectively off by default. Test Any way to avoid the sleep(5000) in the test? Our tests already take a really long time, so anytime we can avoid sleeping it's better. Maybe wait at most 5 seconds for timeWaitingMs.get() to become > 0 directoryScannerThrottle Shouldn't stop call resume() instead of just notifyAll(). Will cycle() get out if we try to shutdown while in that wait? Did we hit this problem with too big of hammer? Couldn't cycle() be implemented with a simple sleep? For example, with a 75% duty cycle, n = Time.monotonicNow() % 1000; if (n > 1000 * 0.75) sleep(1000- n) Seems like it could be as simple as a config and a couple of lines of code. Maybe I'm missing something or there are grander plans for the throttle.
          Hide
          templedf Daniel Templeton added a comment -

          I personally would prefer the default to be 1000.

          I went with 0 because that's the general convention for "off", but if 1000 makes more sense to people it's fine with me.

          Any way to avoid the sleep(5000) in the test?

          I hear ya. I'll see if I can reformulate the test a bit to get the sleep time down to the bare minimum.

          Shouldn't stop call resume() instead of just notifyAll().

          Sure. It won't matter since stop() is only called by shutdown(), which first sets shouldRunCompile to false. But for correctness, you're right.

          Did we hit this problem with too big of hammer?

          The majority of the patch is refactoring the report compilers so that they can be throttled at all. The additional code to do the throttling isn't much. It's more formal than just a sleep, but it's also more testable and extensible.

          Show
          templedf Daniel Templeton added a comment - I personally would prefer the default to be 1000. I went with 0 because that's the general convention for "off", but if 1000 makes more sense to people it's fine with me. Any way to avoid the sleep(5000) in the test? I hear ya. I'll see if I can reformulate the test a bit to get the sleep time down to the bare minimum. Shouldn't stop call resume() instead of just notifyAll(). Sure. It won't matter since stop() is only called by shutdown(), which first sets shouldRunCompile to false. But for correctness, you're right. Did we hit this problem with too big of hammer? The majority of the patch is refactoring the report compilers so that they can be throttled at all. The additional code to do the throttling isn't much. It's more formal than just a sleep, but it's also more testable and extensible.
          Hide
          nroberts Nathan Roberts added a comment -

          Sure. It won't matter since stop() is only called by shutdown(), which first sets shouldRunCompile to false. But for correctness, you're right.

          I was looking at v3 of the patch, I see you already fixed this in v4 of the patch. sorry for the noise

          The majority of the patch is refactoring the report compilers so that they can be throttled at all. The additional code to do the throttling isn't much. It's more formal than just a sleep, but it's also more testable and extensible.

          ok. I'll have to look at that a little deeper. I thought we were basically hitting FileUtil.listFiles(dir) really quickly in the original code so it felt like a very simple thing to do is just do a variable sleep right there based on the configured duty cycle.

          I need to look more into how the scanjob queue is working. It seems like all the worker threads could be working in the same volume which doesn't seem like what we want. (Seems like we want all volumes spending the duty cycle scanning, but I didn't catch how that was the case).

          Show
          nroberts Nathan Roberts added a comment - Sure. It won't matter since stop() is only called by shutdown(), which first sets shouldRunCompile to false. But for correctness, you're right. I was looking at v3 of the patch, I see you already fixed this in v4 of the patch. sorry for the noise The majority of the patch is refactoring the report compilers so that they can be throttled at all. The additional code to do the throttling isn't much. It's more formal than just a sleep, but it's also more testable and extensible. ok. I'll have to look at that a little deeper. I thought we were basically hitting FileUtil.listFiles(dir) really quickly in the original code so it felt like a very simple thing to do is just do a variable sleep right there based on the configured duty cycle. I need to look more into how the scanjob queue is working. It seems like all the worker threads could be working in the same volume which doesn't seem like what we want. (Seems like we want all volumes spending the duty cycle scanning, but I didn't catch how that was the case).
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 54s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 10m 5s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 20s The applied patch generated 6 new checkstyle issues (total was 442, now 427).
          -1 whitespace 0m 6s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 174m 14s Tests failed in hadoop-hdfs.
              220m 3s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
          Timed out tests org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761197/HDFS-8873.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 94dec5a
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12550/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12550/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12550/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12550/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12550/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12550/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 5s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 20s The applied patch generated 6 new checkstyle issues (total was 442, now 427). -1 whitespace 0m 6s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 174m 14s Tests failed in hadoop-hdfs.     220m 3s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure Timed out tests org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761197/HDFS-8873.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 94dec5a checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12550/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12550/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12550/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12550/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12550/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12550/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          The scanjob queue is indeed ignoring volume when selecting the next job. I was considering the case where there are volumes of greatly differing sizes, in which case not binding a thread to a volume will result in a better distribution of the load. That's also true when the number of threads exceeds the number of volumes.

          That said, the point of the JIRA was not to change the load profile of the directory scanner; it was just to insert a throttle. I'll post a changeset with a reduced scope shortly.

          Show
          templedf Daniel Templeton added a comment - The scanjob queue is indeed ignoring volume when selecting the next job. I was considering the case where there are volumes of greatly differing sizes, in which case not binding a thread to a volume will result in a better distribution of the load. That's also true when the number of threads exceeds the number of volumes. That said, the point of the JIRA was not to change the load profile of the directory scanner; it was just to insert a throttle. I'll post a changeset with a reduced scope shortly.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Nathan Roberts, I agree that it might be better to keep the old behavior of finishing one volume in a thread before moving on to the next. It might increase our cache hit rate. I can think of reasons to do the opposite (i.e. spread the load across disks), that might motivate us to add that mode as an option, but it seems better to focus on just throttling in this change.

          Show
          cmccabe Colin P. McCabe added a comment - Nathan Roberts , I agree that it might be better to keep the old behavior of finishing one volume in a thread before moving on to the next. It might increase our cache hit rate. I can think of reasons to do the opposite (i.e. spread the load across disks), that might motivate us to add that mode as an option, but it seems better to focus on just throttling in this change.
          Hide
          templedf Daniel Templeton added a comment -

          Here's a new patch with greatly reduced scope. Colin P. McCabe and Nathan Roberts, please have a look.

          Show
          templedf Daniel Templeton added a comment - Here's a new patch with greatly reduced scope. Colin P. McCabe and Nathan Roberts , please have a look.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Daniel Templeton.

          416	    if ((throttle > 1000) || (throttle <= 0)) {
          417	      if (throttle > 1000) {
          418	        LOG.error(
          419	            DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_THROTTLE_LIMIT_MS_PER_SEC_KEY
          420	            + " set to value above 1000 ms/sec. Assuming default value of " +
          421	            DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_THROTTLE_LIMIT_MS_PER_SEC_DEFAULT);
          

          Can we have a constant here for MS_PER_SEC? I think I commented on this earlier

          455	    if (throttleLimitMsPerSec < 1000) {
          456	      logMsg = String.format("Periodic Directory Tree Verification scan"
          457	          + " starting at %dms with interval %dms and run limit %dms/s",
          458	          firstScanTime, scanPeriodMsecs, throttleLimitMsPerSec);
          

          Maybe say "throttle" instead of "run limit"?

          766	    // Variable for tracking time spent running and waiting for testing
          767	    // purposes
          768	    private Long markMs;
          

          Does this need to be an object, or can it be a primitive? I don't see any case where we need it to be null.

          895	      while (nowMs % 1000L > throttleLimitMsPerSec) {
          896	        try {
          897	          Thread.sleep(1000L - (nowMs % 1000L));
          898	        } catch (InterruptedException ex) {
          899	          // Try sleeping again and mark the thread as interrupted
          900	          Thread.currentThread().interrupt();
          901	        }
          

          This logic seems flawed. If we sleep for a whole second, we'll be back in the case where nowMs % 1000 is what it was before, and make no progress. We should gracefully handle the case where the sleep is longer than a second by not re-throttling.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Daniel Templeton . 416 if ((throttle > 1000) || (throttle <= 0)) { 417 if (throttle > 1000) { 418 LOG.error( 419 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_THROTTLE_LIMIT_MS_PER_SEC_KEY 420 + " set to value above 1000 ms/sec. Assuming default value of " + 421 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_THROTTLE_LIMIT_MS_PER_SEC_DEFAULT); Can we have a constant here for MS_PER_SEC ? I think I commented on this earlier 455 if (throttleLimitMsPerSec < 1000) { 456 logMsg = String .format( "Periodic Directory Tree Verification scan" 457 + " starting at %dms with interval %dms and run limit %dms/s" , 458 firstScanTime, scanPeriodMsecs, throttleLimitMsPerSec); Maybe say "throttle" instead of "run limit"? 766 // Variable for tracking time spent running and waiting for testing 767 // purposes 768 private Long markMs; Does this need to be an object, or can it be a primitive? I don't see any case where we need it to be null. 895 while (nowMs % 1000L > throttleLimitMsPerSec) { 896 try { 897 Thread .sleep(1000L - (nowMs % 1000L)); 898 } catch (InterruptedException ex) { 899 // Try sleeping again and mark the thread as interrupted 900 Thread .currentThread().interrupt(); 901 } This logic seems flawed. If we sleep for a whole second, we'll be back in the case where nowMs % 1000 is what it was before, and make no progress. We should gracefully handle the case where the sleep is longer than a second by not re-throttling.
          Hide
          templedf Daniel Templeton added a comment -

          Can we have a constant here for MS_PER_SEC? I think I commented on this earlier

          I didn't do that in this patch because I didn't think the 1000 was as prominent as before, but it appears that was before I was done adding stuff. I'll put it back. I'd love to put that constant somewhere like util.Time. Would that be kosher? Or better keep it low profile and leave it local to DirectoryScanner? I notice there's already HdfsClientConfigKeys.SECOND, but that would introduce an pointless dependency. May the best answer is to keep it local and file a JIRA to consolidate them under util.Time?

          Maybe say "throttle" instead of "run limit"?

          I was shooting for something that would be meaningful to someone who doesn't know the code. What about "throttle limit," since that echoes the config param?

          Does this need to be an object, or can it be a primitive?

          Ha. Evolutionary mistake. I'll fix it.

          This logic seems flawed.

          I don't follow. (nowMs % 1000) has to be between 0 and 999. If it's less than the throttle limit, we won't enter the loop. The throttle limit must be between 1 and 1000. (Anything else gets set to 1000 when the scanner is created.) The sleep must therefore be for between 1 and 999 ms, pretty much guaranteeing a different result the next time around.

          Show
          templedf Daniel Templeton added a comment - Can we have a constant here for MS_PER_SEC? I think I commented on this earlier I didn't do that in this patch because I didn't think the 1000 was as prominent as before, but it appears that was before I was done adding stuff. I'll put it back. I'd love to put that constant somewhere like util.Time. Would that be kosher? Or better keep it low profile and leave it local to DirectoryScanner? I notice there's already HdfsClientConfigKeys.SECOND, but that would introduce an pointless dependency. May the best answer is to keep it local and file a JIRA to consolidate them under util.Time? Maybe say "throttle" instead of "run limit"? I was shooting for something that would be meaningful to someone who doesn't know the code. What about "throttle limit," since that echoes the config param? Does this need to be an object, or can it be a primitive? Ha. Evolutionary mistake. I'll fix it. This logic seems flawed. I don't follow. (nowMs % 1000) has to be between 0 and 999. If it's less than the throttle limit, we won't enter the loop. The throttle limit must be between 1 and 1000. (Anything else gets set to 1000 when the scanner is created.) The sleep must therefore be for between 1 and 999 ms, pretty much guaranteeing a different result the next time around.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I didn't do that in this patch because I didn't think the 1000 was as prominent as before, but it appears that was before I was done adding stuff. I'll put it back. I'd love to put that constant somewhere like util.Time. Would that be kosher?

          I think it's fine to put it in the DirectoryScanner itself if you want. I don't object to putting it in time either. Up to you.

          I was shooting for something that would be meaningful to someone who doesn't know the code. What about "throttle limit," since that echoes the config param?

          Sure.

          I don't follow. (nowMs % 1000) has to be between 0 and 999. If it's less than the throttle limit, we won't enter the loop. The throttle limit must be between 1 and 1000. (Anything else gets set to 1000 when the scanner is created.) The sleep must therefore be for between 1 and 999 ms, pretty much guaranteeing a different result the next time around.

          Let's say we start the loop at time 5200. Then while (nowMs % 1000L > throttleLimitMsPerSec) returns true (let's say throttleLimitMsPerSec = 100).

          We call sleep with an argument of 800, but sleep actually sleeps for 1000 ms instead. (Remember, Thread#sleep may always sleep for longer than requested.) nowMs becomes 6200. Now {{while (nowMs % 1000L > throttleLimitMsPerSec) }} returns true again, since 6200 % 1000 = 200 > 100. Now we sleep again for 800 ms yet again. We completely missed our timeslice, and there's no guarantee that we'll pick up the next one either. That's the bug.

          Show
          cmccabe Colin P. McCabe added a comment - I didn't do that in this patch because I didn't think the 1000 was as prominent as before, but it appears that was before I was done adding stuff. I'll put it back. I'd love to put that constant somewhere like util.Time. Would that be kosher? I think it's fine to put it in the DirectoryScanner itself if you want. I don't object to putting it in time either. Up to you. I was shooting for something that would be meaningful to someone who doesn't know the code. What about "throttle limit," since that echoes the config param? Sure. I don't follow. (nowMs % 1000) has to be between 0 and 999. If it's less than the throttle limit, we won't enter the loop. The throttle limit must be between 1 and 1000. (Anything else gets set to 1000 when the scanner is created.) The sleep must therefore be for between 1 and 999 ms, pretty much guaranteeing a different result the next time around. Let's say we start the loop at time 5200. Then while (nowMs % 1000L > throttleLimitMsPerSec) returns true (let's say throttleLimitMsPerSec = 100 ). We call sleep with an argument of 800, but sleep actually sleeps for 1000 ms instead. (Remember, Thread#sleep may always sleep for longer than requested.) nowMs becomes 6200. Now {{while (nowMs % 1000L > throttleLimitMsPerSec) }} returns true again, since 6200 % 1000 = 200 > 100. Now we sleep again for 800 ms yet again. We completely missed our timeslice, and there's no guarantee that we'll pick up the next one either. That's the bug.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 59s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 52s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 20s The applied patch generated 8 new checkstyle issues (total was 442, now 443).
          -1 whitespace 0m 2s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          +1 hdfs tests 178m 37s Tests passed in hadoop-hdfs.
              223m 56s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761689/HDFS-8873.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cc2b473
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12608/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12608/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12608/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12608/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12608/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 59s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 20s The applied patch generated 8 new checkstyle issues (total was 442, now 443). -1 whitespace 0m 2s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion +1 hdfs tests 178m 37s Tests passed in hadoop-hdfs.     223m 56s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761689/HDFS-8873.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cc2b473 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12608/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12608/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12608/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12608/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12608/console This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Thanks Daniel Templeton for the update. I am sorry I haven't had a chance to review yet. I plan to do this Thursday AM.

          Show
          nroberts Nathan Roberts added a comment - Thanks Daniel Templeton for the update. I am sorry I haven't had a chance to review yet. I plan to do this Thursday AM.
          Hide
          templedf Daniel Templeton added a comment -

          No worries, Nathan Roberts. I'm about to upload a new patch anyway.

          Show
          templedf Daniel Templeton added a comment - No worries, Nathan Roberts . I'm about to upload a new patch anyway.
          Hide
          templedf Daniel Templeton added a comment -

          New patch to address Colin P. McCabe's comments.

          Based on the results of previous patch, this one will get flagged with 3 whitespace violations in lines I didn't touch and a fist full of checkstyle issues because of the length of the config key names.

          Show
          templedf Daniel Templeton added a comment - New patch to address Colin P. McCabe 's comments. Based on the results of previous patch, this one will get flagged with 3 whitespace violations in lines I didn't touch and a fist full of checkstyle issues because of the length of the config key names.
          Hide
          templedf Daniel Templeton added a comment -

          Forgot to mention that this patch tries be fair about oversleeps when throttling, but it doesn't do anything to credit for full seconds lost entirely to oversleeps. For example, if the throttle is set to 100ms, the following could happen:

          .0000 - thread calls throttle(), no block
          .0422 - thread calls throttle(), sleep for 588ms
          .5016 - thread wakes up from oversleep, run limit this second set to 116ms
          .5219 - thread calls throttle(), sleep for 781ms

          In the second that the thread wakes up from the oversleep, we try to give it its full run time, but we don't credit it for full seconds lost. Notice also that threads only offer to block once per cycle, so setting a very low throttle limit will just make the threads run one cycle between sleeps.

          The following could also happen:

          .0000 - thread calls throttle(), no block
          .0422 - thread calls throttle(), sleep for 588ms
          .1970 - thread wakes up from oversleep, run limit this second set to 1070ms
          .2089 - thread calls throttle(), no block
          .2101 - thread calls throttle(), sleep for 899ms

          When the thread wakes up from the oversleep, we give it more than a second's worth of run time because it's so close to the end of the second. (Any value over 999 just means not to throttle the thread that second.) The next second starts the count over, and at .2089, even though it's been running for over 100 consecutive ms, we don't block it, because it hasn't run for 100ms in this second yet. When it checks in the next time, at .2101, we see it's over the 100ms limit and block it for the rest of the second.

          Show
          templedf Daniel Templeton added a comment - Forgot to mention that this patch tries be fair about oversleeps when throttling, but it doesn't do anything to credit for full seconds lost entirely to oversleeps. For example, if the throttle is set to 100ms, the following could happen: .0000 - thread calls throttle(), no block .0422 - thread calls throttle(), sleep for 588ms .5016 - thread wakes up from oversleep, run limit this second set to 116ms .5219 - thread calls throttle(), sleep for 781ms In the second that the thread wakes up from the oversleep, we try to give it its full run time, but we don't credit it for full seconds lost. Notice also that threads only offer to block once per cycle, so setting a very low throttle limit will just make the threads run one cycle between sleeps. The following could also happen: .0000 - thread calls throttle(), no block .0422 - thread calls throttle(), sleep for 588ms .1970 - thread wakes up from oversleep, run limit this second set to 1070ms .2089 - thread calls throttle(), no block .2101 - thread calls throttle(), sleep for 899ms When the thread wakes up from the oversleep, we give it more than a second's worth of run time because it's so close to the end of the second. (Any value over 999 just means not to throttle the thread that second.) The next second starts the count over, and at .2089, even though it's been running for over 100 consecutive ms, we don't block it, because it hasn't run for 100ms in this second yet. When it checks in the next time, at .2101, we see it's over the 100ms limit and block it for the rest of the second.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 15s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 10m 15s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 24s The applied patch generated 8 new checkstyle issues (total was 442, now 443).
          -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 16s Pre-build of native portion
          -1 hdfs tests 138m 33s Tests failed in hadoop-hdfs.
              184m 47s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestDFSRename
            hadoop.hdfs.TestDatanodeConfig
            hadoop.hdfs.TestBlockReaderFactory
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.TestSnapshotCommands
            hadoop.hdfs.TestFileConcurrentReader
            hadoop.hdfs.TestFileAppend2
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestLeaseRecovery
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.tools.TestDelegationTokenFetcher
            hadoop.tracing.TestTraceAdmin
            hadoop.hdfs.TestWriteConfigurationToDFS
            hadoop.hdfs.tools.TestStoragePolicyCommands
            hadoop.hdfs.TestReplication
            hadoop.hdfs.TestPipelines
            hadoop.hdfs.tools.TestGetGroups
            hadoop.hdfs.TestParallelShortCircuitRead
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.hdfs.TestBlocksScheduledCounter
            hadoop.hdfs.TestRollingUpgradeRollback
            hadoop.hdfs.protocolPB.TestPBHelper
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.TestFSInputChecker
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestRead
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.TestHDFSTrash
            hadoop.hdfs.TestExternalBlockReader
            hadoop.hdfs.TestDatanodeRegistration
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
            hadoop.hdfs.TestSmallBlock
            hadoop.hdfs.TestDFSRollback
            hadoop.hdfs.TestDFSStorageStateRecovery
            hadoop.hdfs.TestDFSUtil
            hadoop.hdfs.TestHdfsAdmin
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
            hadoop.hdfs.TestAbandonBlock
            hadoop.hdfs.TestConnCache
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.TestHDFSServerPorts
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.TestRollingUpgradeDowngrade
            hadoop.hdfs.TestDFSFinalize
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.TestBlockMissingException
            hadoop.hdfs.TestFileAppend3
            hadoop.hdfs.TestFileLengthOnClusterRestart
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.tools.TestDebugAdmin
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.TestDFSPermission
            hadoop.hdfs.TestDFSInotifyEventInputStream
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestDFSClientExcludedNodes
            hadoop.hdfs.TestMiniDFSCluster
            hadoop.hdfs.TestParallelRead
            hadoop.hdfs.TestPread
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestReservedRawPaths
            hadoop.hdfs.TestEncryptionZonesWithHA
            hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.TestParallelUnixDomainRead
            hadoop.hdfs.TestDatanodeReport
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestDatanodeLayoutUpgrade
            hadoop.hdfs.TestFileAppendRestart
            hadoop.hdfs.TestRemoteBlockReader2
            hadoop.hdfs.TestSetrepIncreasing
            hadoop.hdfs.security.TestDelegationToken
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestReadWhileWriting
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
            hadoop.hdfs.TestLease
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.TestFetchImage
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.hdfs.TestClientBlockVerification
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.TestWriteBlockGetsBlockLengthHint
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.hdfs.security.token.block.TestBlockToken
            hadoop.hdfs.tools.TestDFSZKFailoverController
            hadoop.hdfs.TestBlockReaderLocal
            hadoop.hdfs.TestHDFSFileSystemContract
            hadoop.hdfs.TestDFSStartupVersions
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestDatanodeDeath
            hadoop.hdfs.TestSafeMode
            hadoop.hdfs.TestListFilesInFileContext
            hadoop.hdfs.TestSeekBug
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
            hadoop.hdfs.TestDisableConnCache
            hadoop.hdfs.TestLargeBlock
            hadoop.hdfs.TestBlockReaderLocalLegacy
            hadoop.hdfs.TestMultiThreadedHflush
            hadoop.tracing.TestTracingShortCircuitLocalRead
            hadoop.hdfs.TestClose
            hadoop.hdfs.TestSetTimes



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761918/HDFS-8873.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c890c51
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12625/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12625/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12625/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12625/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12625/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 15s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 24s The applied patch generated 8 new checkstyle issues (total was 442, now 443). -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 16s Pre-build of native portion -1 hdfs tests 138m 33s Tests failed in hadoop-hdfs.     184m 47s   Reason Tests Failed unit tests hadoop.hdfs.TestDFSRename   hadoop.hdfs.TestDatanodeConfig   hadoop.hdfs.TestBlockReaderFactory   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.TestSnapshotCommands   hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.tracing.TestTraceAdmin   hadoop.hdfs.TestWriteConfigurationToDFS   hadoop.hdfs.tools.TestStoragePolicyCommands   hadoop.hdfs.TestReplication   hadoop.hdfs.TestPipelines   hadoop.hdfs.tools.TestGetGroups   hadoop.hdfs.TestParallelShortCircuitRead   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.TestBlocksScheduledCounter   hadoop.hdfs.TestRollingUpgradeRollback   hadoop.hdfs.protocolPB.TestPBHelper   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.TestFSInputChecker   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestRead   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.TestExternalBlockReader   hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.tools.TestDFSHAAdminMiniCluster   hadoop.hdfs.TestSmallBlock   hadoop.hdfs.TestDFSRollback   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.hdfs.TestDFSUtil   hadoop.hdfs.TestHdfsAdmin   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary   hadoop.hdfs.TestAbandonBlock   hadoop.hdfs.TestConnCache   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.TestHDFSServerPorts   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.TestRollingUpgradeDowngrade   hadoop.hdfs.TestDFSFinalize   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestBlockMissingException   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.tools.TestDebugAdmin   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestDFSPermission   hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestDFSClientExcludedNodes   hadoop.hdfs.TestMiniDFSCluster   hadoop.hdfs.TestParallelRead   hadoop.hdfs.TestPread   hadoop.tracing.TestTracing   hadoop.hdfs.TestReservedRawPaths   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.TestHFlush   hadoop.hdfs.TestParallelUnixDomainRead   hadoop.hdfs.TestDatanodeReport   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.TestRemoteBlockReader2   hadoop.hdfs.TestSetrepIncreasing   hadoop.hdfs.security.TestDelegationToken   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestReadWhileWriting   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr   hadoop.hdfs.TestLease   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestFetchImage   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.TestClientBlockVerification   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.TestWriteBlockGetsBlockLengthHint   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.security.token.block.TestBlockToken   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.TestDFSStartupVersions   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestDatanodeDeath   hadoop.hdfs.TestSafeMode   hadoop.hdfs.TestListFilesInFileContext   hadoop.hdfs.TestSeekBug   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl   hadoop.hdfs.TestDisableConnCache   hadoop.hdfs.TestLargeBlock   hadoop.hdfs.TestBlockReaderLocalLegacy   hadoop.hdfs.TestMultiThreadedHflush   hadoop.tracing.TestTracingShortCircuitLocalRead   hadoop.hdfs.TestClose   hadoop.hdfs.TestSetTimes Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761918/HDFS-8873.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c890c51 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12625/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12625/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12625/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12625/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12625/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          I guess the short summary is that I'm not happy with the "modulo" solution to the timekeeping problem since I think its inaccuracy (in the direction of not running often enough, or at all) will be unacceptable at lower throttle rates.

          We have a StopWatch class which should provide a pretty easy solution to this problem that doesn't suffer from these issues. Just start a StopWatch when you enable scanning. Check the elapsed duration on the StopWatch before scanning each block-- if it is above dfs.datanode.directoryscan.throttle.limit.ms.per.sec, then disable scanning and go into the other loop, which calls Thread.sleep until 1000 - dfs.datanode.directoryscan.throttle.limit.ms.per.sec ms have elapsed.

          This solution is robust against slightly shorter or longer thread pauses. Like the modulo solution, it also may run at a slightly lower rate than expected in the case where we oversleep. Unlike the modulo solution, it does not have a pathological case where we never get to run at all despite a non-zero throttle rate.

          399  public static final int     DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT = 21600;
          ...
          460	    int defaultLimit =
          461	        DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT;
          462	    String logMsg;
          463	
          464	    if (throttleLimitMsPerSec < defaultLimit) {
          465	      logMsg = String.format(START_MESSAGE_WITH_THROTTLE, firstScanTime,
          466	          scanPeriodMsecs, throttleLimitMsPerSec);
          467	    } else {
          468	      logMsg = String.format(START_MESSAGE, firstScanTime, scanPeriodMsecs);
          469	    }
          

          I'm confused about this code... why are we comparing the directory scan interval (in seconds) with the throttle limit in milliseconds? It seems like this was intended to test for the case where the throttle is disabled, but some wires got crossed?

          Show
          cmccabe Colin P. McCabe added a comment - - edited I guess the short summary is that I'm not happy with the "modulo" solution to the timekeeping problem since I think its inaccuracy (in the direction of not running often enough, or at all) will be unacceptable at lower throttle rates. We have a StopWatch class which should provide a pretty easy solution to this problem that doesn't suffer from these issues. Just start a StopWatch when you enable scanning. Check the elapsed duration on the StopWatch before scanning each block-- if it is above dfs.datanode.directoryscan.throttle.limit.ms.per.sec, then disable scanning and go into the other loop, which calls Thread.sleep until 1000 - dfs.datanode.directoryscan.throttle.limit.ms.per.sec ms have elapsed. This solution is robust against slightly shorter or longer thread pauses. Like the modulo solution, it also may run at a slightly lower rate than expected in the case where we oversleep. Unlike the modulo solution, it does not have a pathological case where we never get to run at all despite a non-zero throttle rate. 399 public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT = 21600; ... 460 int defaultLimit = 461 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT; 462 String logMsg; 463 464 if (throttleLimitMsPerSec < defaultLimit) { 465 logMsg = String .format(START_MESSAGE_WITH_THROTTLE, firstScanTime, 466 scanPeriodMsecs, throttleLimitMsPerSec); 467 } else { 468 logMsg = String .format(START_MESSAGE, firstScanTime, scanPeriodMsecs); 469 } I'm confused about this code... why are we comparing the directory scan interval (in seconds) with the throttle limit in milliseconds? It seems like this was intended to test for the case where the throttle is disabled, but some wires got crossed?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 9s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 10s There were no new javac warning messages.
          +1 javadoc 10m 18s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 28s The applied patch generated 8 new checkstyle issues (total was 442, now 443).
          -1 whitespace 0m 2s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 2m 37s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 21s Pre-build of native portion
          -1 hdfs tests 194m 6s Tests failed in hadoop-hdfs.
              240m 55s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.server.blockmanagement.TestNodeCount



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761923/HDFS-8873.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c890c51
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12626/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12626/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12626/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12626/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12626/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 10s There were no new javac warning messages. +1 javadoc 10m 18s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 28s The applied patch generated 8 new checkstyle issues (total was 442, now 443). -1 whitespace 0m 2s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 2m 37s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 21s Pre-build of native portion -1 hdfs tests 194m 6s Tests failed in hadoop-hdfs.     240m 55s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.blockmanagement.TestNodeCount Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761923/HDFS-8873.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c890c51 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12626/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12626/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12626/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12626/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12626/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Unlike the modulo solution, it does not have a pathological case where we never get to run at all despite a non-zero throttle rate.

          I don't see the pathological case. Assume the throttle is 1ms and we always oversleep:

          .0000 - thread calls throttle(), no block
          – thread runs –
          .0422 - thread calls throttle(), sleep for 588ms
          .1999 - thread wakes up from oversleep, run limit this second set to 1000ms
          – thread runs (999 < 1000) –
          .2190 - thread calls throttle(), sleep for 810ms
          .3106 - thread wakes up from oversleep, run limit this second set to 107ms
          – thread runs (106 < 107) –
          etc.

          The throttle() method is guaranteed to exit when at least (1000 - limit) ms have passed and the calling thread is scheduled again. What am I missing?

          As fas as I can tell, the difference between the StopWatch approach and the modulo approach is the case when a thread wakes up within (1000 - limit) of the end of the second. In that case, the modulo approach will allow the thread to run longer than the StopWatch approach would. In other words, the modulo approach is focused on trying to hold to the per-second duty cycle, and the StopWatch approach is trying to hold to the duty cycle regardless of second boundaries. Given that neither the sleep time nor run time is reliable, I don't see where it matters much one way or the other.

          That said, I did just notice that this patch is broken in the case of spurious wake-ups. I'll have a new patch for that (including tests) shortly.

          I'm confused about this code...

          Good catch. Copy-paste error, grabbed the wrong key.

          Show
          templedf Daniel Templeton added a comment - Unlike the modulo solution, it does not have a pathological case where we never get to run at all despite a non-zero throttle rate. I don't see the pathological case. Assume the throttle is 1ms and we always oversleep: .0000 - thread calls throttle(), no block – thread runs – .0422 - thread calls throttle(), sleep for 588ms .1999 - thread wakes up from oversleep, run limit this second set to 1000ms – thread runs (999 < 1000) – .2190 - thread calls throttle(), sleep for 810ms .3106 - thread wakes up from oversleep, run limit this second set to 107ms – thread runs (106 < 107) – etc. The throttle() method is guaranteed to exit when at least (1000 - limit) ms have passed and the calling thread is scheduled again. What am I missing? As fas as I can tell, the difference between the StopWatch approach and the modulo approach is the case when a thread wakes up within (1000 - limit) of the end of the second. In that case, the modulo approach will allow the thread to run longer than the StopWatch approach would. In other words, the modulo approach is focused on trying to hold to the per-second duty cycle, and the StopWatch approach is trying to hold to the duty cycle regardless of second boundaries. Given that neither the sleep time nor run time is reliable, I don't see where it matters much one way or the other. That said, I did just notice that this patch is broken in the case of spurious wake-ups. I'll have a new patch for that (including tests) shortly. I'm confused about this code... Good catch. Copy-paste error, grabbed the wrong key.
          Hide
          templedf Daniel Templeton added a comment -

          OK, here's a version reworked to use the StopWatch class. After a little research, I think we're safe to trust that Thread.sleep() won't have spurious wake-ups, which makes the StopWatch code simpler than the modulo version. If we want to care about spurious wake-ups, then the modulo code is simpler.

          This patch is still failing one of the new tests I just added. I will have to fix that, but I wanted to get this patch posted for review by Nathan Roberts this morning. I'll post a fixed patch when I get a chance.

          Show
          templedf Daniel Templeton added a comment - OK, here's a version reworked to use the StopWatch class. After a little research, I think we're safe to trust that Thread.sleep() won't have spurious wake-ups, which makes the StopWatch code simpler than the modulo version. If we want to care about spurious wake-ups, then the modulo code is simpler. This patch is still failing one of the new tests I just added. I will have to fix that, but I wanted to get this patch posted for review by Nathan Roberts this morning. I'll post a fixed patch when I get a chance.
          Hide
          templedf Daniel Templeton added a comment -

          All the tests pass now.

          Show
          templedf Daniel Templeton added a comment - All the tests pass now.
          Hide
          nroberts Nathan Roberts added a comment -

          Thanks Daniel Templeton. I like that the stopwatch class makes this much cleaner. Just a couple of comments:

          • Shouldn't the isInterrupted() check throw an InterruptedException? Otherwise won't we just break out of one level? It would probably be good to test shutdown on an actual cluster if possible because you're exactly right that we could be in here a long time and it would be good to make sure we don't affect shutdown of the datanode. This has been a problem in the past and can have a serious impact on rolling upgrades.
          • nit but I find markRunning() and markWaiting() confusing (seem backwards to me because we call markRunning() just before going to sleep).
          • I'm kind of wondering if we should disallow extremely low duty cycles. Seems like it could take close to 24 hours with a minimum setting. A minimum of 20% should keep us within an hour.
          Show
          nroberts Nathan Roberts added a comment - Thanks Daniel Templeton . I like that the stopwatch class makes this much cleaner. Just a couple of comments: Shouldn't the isInterrupted() check throw an InterruptedException? Otherwise won't we just break out of one level? It would probably be good to test shutdown on an actual cluster if possible because you're exactly right that we could be in here a long time and it would be good to make sure we don't affect shutdown of the datanode. This has been a problem in the past and can have a serious impact on rolling upgrades. nit but I find markRunning() and markWaiting() confusing (seem backwards to me because we call markRunning() just before going to sleep). I'm kind of wondering if we should disallow extremely low duty cycles. Seems like it could take close to 24 hours with a minimum setting. A minimum of 20% should keep us within an hour.
          Hide
          templedf Daniel Templeton added a comment -

          Shouldn't the isInterrupted() check throw an InterruptedException?

          In patch 8 it does.

          nit but I find markRunning() and markWaiting() confusing

          I think it will be confusing either way. Maybe rename them logTimeRunning() and logTimeWaiting()?

          I'm kind of wondering if we should disallow extremely low duty cycles.

          I'm kinda caveat emptor on that one. In no case can they shutdown the scanning completely, but if they want to make it take forever, that's their business. I also say that because at this point we don't know what reasonable lower bounds are. Maybe after the patch goes in, you can play with it on your system and tell us what impact various throttle values levels are? We could then follow up with a JIRA to update the docs and maybe bounds checking accordingly.

          Show
          templedf Daniel Templeton added a comment - Shouldn't the isInterrupted() check throw an InterruptedException? In patch 8 it does. nit but I find markRunning() and markWaiting() confusing I think it will be confusing either way. Maybe rename them logTimeRunning() and logTimeWaiting()? I'm kind of wondering if we should disallow extremely low duty cycles. I'm kinda caveat emptor on that one. In no case can they shutdown the scanning completely, but if they want to make it take forever, that's their business. I also say that because at this point we don't know what reasonable lower bounds are. Maybe after the patch goes in, you can play with it on your system and tell us what impact various throttle values levels are? We could then follow up with a JIRA to update the docs and maybe bounds checking accordingly.
          Hide
          templedf Daniel Templeton added a comment -

          I renamed the markXing() methods to accumulateTimeXing(). I also fixed some javadoc issues I missed last time.

          Show
          templedf Daniel Templeton added a comment - I renamed the markXing() methods to accumulateTimeXing(). I also fixed some javadoc issues I missed last time.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 54s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 1s There were no new javac warning messages.
          +1 javadoc 10m 12s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 25s The applied patch generated 8 new checkstyle issues (total was 439, now 440).
          -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          -1 findbugs 2m 38s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 83m 54s Tests failed in hadoop-hdfs.
              129m 56s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Timed out tests org.apache.hadoop.hdfs.TestFileCreation



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762164/HDFS-8873.007.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 71a81b6
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12660/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12660/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12660/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12660/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12660/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12660/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 12s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 8 new checkstyle issues (total was 439, now 440). -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 2m 38s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 83m 54s Tests failed in hadoop-hdfs.     129m 56s   Reason Tests FindBugs module:hadoop-hdfs Timed out tests org.apache.hadoop.hdfs.TestFileCreation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762164/HDFS-8873.007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71a81b6 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12660/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12660/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12660/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12660/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12660/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12660/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 48s There were no new javac warning messages.
          +1 javadoc 10m 27s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 25s The applied patch generated 8 new checkstyle issues (total was 439, now 439).
          -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 36s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 49m 46s Tests failed in hadoop-hdfs.
              95m 33s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestFileStatus
            hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
            hadoop.hdfs.server.namenode.TestINodeFile
            hadoop.fs.contract.hdfs.TestHDFSContractOpen
            hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
            hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.server.namenode.ha.TestHASafeMode
            hadoop.hdfs.TestEncryptionZonesWithHA
            hadoop.fs.contract.hdfs.TestHDFSContractMkdir
            hadoop.hdfs.TestDFSShell
            hadoop.hdfs.server.namenode.TestNameNodeXAttr
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDecommission
            hadoop.hdfs.server.namenode.TestFSEditLogLoader
            hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.cli.TestXAttrCLI
            hadoop.hdfs.server.namenode.TestDeleteRace
            hadoop.hdfs.server.namenode.TestParallelImageWrite
            hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys
            hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional
            hadoop.hdfs.server.namenode.TestSaveNamespace
            hadoop.hdfs.TestDFSRename
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
            hadoop.hdfs.server.datanode.TestDeleteBlockPool
            hadoop.hdfs.TestRemoteBlockReader2
            hadoop.hdfs.server.namenode.TestStorageRestore
            hadoop.hdfs.server.namenode.TestFileLimit
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.TestEncryptionZones
            hadoop.fs.contract.hdfs.TestHDFSContractSetTimes
            hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
            hadoop.hdfs.qjournal.TestNNWithQJM
            hadoop.hdfs.tools.TestDelegationTokenFetcher
            hadoop.hdfs.web.TestWebHdfsFileSystemContract
            hadoop.hdfs.server.namenode.TestSecureNameNode
            hadoop.hdfs.server.namenode.TestFileContextAcl
            hadoop.hdfs.TestDatanodeRegistration
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.TestDataTransferProtocol
            hadoop.fs.viewfs.TestViewFsWithXAttrs
            hadoop.hdfs.server.datanode.TestDiskError
            hadoop.hdfs.TestFsShellPermission
            hadoop.hdfs.server.namenode.TestMalformedURLs
            hadoop.hdfs.TestBalancerBandwidth
            hadoop.hdfs.TestReadWhileWriting
            hadoop.fs.TestSWebHdfsFileContextMainOperations
            hadoop.hdfs.TestDisableConnCache
            hadoop.hdfs.TestIsMethodSupported
            hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            hadoop.hdfs.server.blockmanagement.TestAvailableSpaceBlockPlacementPolicy
            hadoop.hdfs.TestFileCreationClient
            hadoop.cli.TestAclCLI
            hadoop.hdfs.TestFSOutputSummer
            hadoop.fs.TestHDFSFileContextMainOperations
            hadoop.hdfs.TestDFSStartupVersions
            hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure
            hadoop.hdfs.server.namenode.TestGenericJournalConf
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestFileAppend2
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
            hadoop.hdfs.server.datanode.TestTransferRbw
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.TestGetFileChecksum
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestFSImageWithAcl
            hadoop.fs.TestUnbuffer
            hadoop.hdfs.TestDFSPermission
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold
            hadoop.hdfs.server.namenode.TestMetaSave
            hadoop.hdfs.TestSmallBlock
            hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
            hadoop.hdfs.web.TestHttpsFileSystem
            hadoop.tracing.TestTracing
            hadoop.hdfs.server.datanode.TestTriggerBlockReport
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
            hadoop.fs.contract.hdfs.TestHDFSContractConcat
            hadoop.hdfs.TestReplication
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.security.TestRefreshUserMappings
            hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.tools.TestDFSZKFailoverController
            hadoop.hdfs.server.namenode.TestHDFSConcat
            hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.namenode.TestFSImageWithXAttr
            hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
            hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.hdfs.TestDFSUtil
            hadoop.security.TestPermissionSymlinks
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.fs.TestSymlinkHdfsDisable
            hadoop.hdfs.server.namenode.TestQuotaByStorageType
            hadoop.hdfs.TestFileCreation
            hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
            hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
            hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
            hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
            hadoop.hdfs.TestRollingUpgradeDowngrade
            hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover
            hadoop.hdfs.TestBlockReaderFactory
            hadoop.hdfs.TestDFSOutputStream
            hadoop.fs.TestUrlStreamHandler
            hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
            hadoop.hdfs.TestModTime
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.tools.TestGetGroups
            hadoop.hdfs.server.namenode.TestNameNodeRecovery
            hadoop.hdfs.TestRestartDFS
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.TestInjectionForSimulatedStorage
            hadoop.hdfs.TestMiniDFSCluster
            hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.TestLease
            hadoop.hdfs.server.namenode.TestProtectedDirectories
            hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
            hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.namenode.TestAuditLogs
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.fs.TestWebHdfsFileContextMainOperations
            hadoop.hdfs.server.namenode.ha.TestNNHealthCheck
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.TestDFSAddressConfig
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
            hadoop.tools.TestJMXGet
            hadoop.hdfs.server.namenode.TestEditLogJournalFailures
            hadoop.hdfs.TestDFSMkdirs
            hadoop.hdfs.TestWriteRead
            hadoop.hdfs.TestClientBlockVerification
            hadoop.fs.viewfs.TestViewFsDefaultValue
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.hdfs.TestDatanodeConfig
            hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
            hadoop.hdfs.server.datanode.TestDataNodeECN
            hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.hdfs.server.datanode.TestDataNodeMXBean
            hadoop.hdfs.server.namenode.TestAclConfigFlag
            hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
            hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
            hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
            hadoop.hdfs.server.namenode.TestAllowFormat
            hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
            hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
            hadoop.hdfs.server.namenode.TestNameNodeRpcServer
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
            hadoop.fs.contract.hdfs.TestHDFSContractDelete
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.server.namenode.TestCheckpoint
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
            hadoop.hdfs.server.namenode.TestDeadDatanode
            hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter
            hadoop.hdfs.TestDatanodeReport
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.TestAuditLogger
            hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.hdfs.TestDFSShellGenericOptions
            hadoop.security.TestPermission
            hadoop.hdfs.web.TestWebHDFSAcl
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.fs.viewfs.TestViewFsHdfs
            hadoop.hdfs.TestClose
            hadoop.cli.TestDeleteCLI
            hadoop.hdfs.tools.TestDebugAdmin
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes
            hadoop.fs.TestFcHdfsCreateMkdir
            hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
            hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.server.namenode.TestXAttrConfigFlag
            hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.fs.permission.TestStickyBit
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.fs.viewfs.TestViewFsWithAcls
            hadoop.hdfs.tools.TestStoragePolicyCommands
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.TestMultiThreadedHflush
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.TestGenericRefresh
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestQuota
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
            hadoop.fs.viewfs.TestViewFsAtHdfsRoot
            hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.tracing.TestTracingShortCircuitLocalRead
            hadoop.fs.TestFcHdfsPermission
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
            hadoop.hdfs.TestLeaseRecovery
            hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
            hadoop.hdfs.TestFileLengthOnClusterRestart
            hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
            hadoop.fs.contract.hdfs.TestHDFSContractCreate
            hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.web.TestWebHDFSOAuth2
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.fs.TestResolveHdfsSymlink
            hadoop.hdfs.server.datanode.TestStorageReport
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.namenode.TestNameNodeMXBean
            hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.tracing.TestTraceAdmin
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename
            hadoop.hdfs.server.namenode.TestNameEditsConfigs
            hadoop.cli.TestCacheAdminCLI
            hadoop.hdfs.server.datanode.TestDataNodeExit
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication
            hadoop.hdfs.server.datanode.TestRefreshNamenodes
            hadoop.hdfs.web.TestWebHDFSForHA
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
            hadoop.hdfs.server.namenode.TestHostsFiles
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.namenode.TestEditLogRace
            hadoop.hdfs.server.datanode.web.webhdfs.TestParameterParser
            hadoop.TestRefreshCallQueue
            hadoop.hdfs.TestSetTimes
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
            hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.fs.TestEnhancedByteBufferAccess
            hadoop.cli.TestCryptoAdminCLI
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.TestDFSStorageStateRecovery
            hadoop.hdfs.TestPersistBlocks
            hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
            hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy
            hadoop.hdfs.TestListFilesInFileContext
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.TestBlockMissingException
            hadoop.hdfs.server.namenode.TestSnapshotPathINodes
            hadoop.hdfs.server.datanode.web.dtp.TestDtpHttp2
            hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
            hadoop.fs.viewfs.TestViewFileSystemHdfs
            hadoop.hdfs.server.datanode.TestBlockRecovery
            hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
            hadoop.hdfs.security.TestDelegationToken
            hadoop.hdfs.server.namenode.TestBlockUnderConstruction
            hadoop.hdfs.TestFileConcurrentReader
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.namenode.TestINodeAttributeProvider
            hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
            hadoop.hdfs.TestParallelShortCircuitReadUnCached
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestRead
            hadoop.hdfs.TestDFSInotifyEventInputStream
            hadoop.hdfs.server.datanode.TestDataNodeInitStorage
            hadoop.hdfs.server.namenode.TestFSNamesystemMBean
            hadoop.fs.loadGenerator.TestLoadGenerator
            hadoop.fs.contract.hdfs.TestHDFSContractAppend
            hadoop.hdfs.server.namenode.TestSecondaryWebUi
            hadoop.fs.viewfs.TestViewFileSystemWithAcls
            hadoop.hdfs.web.TestWebHDFSXAttr
            hadoop.hdfs.server.datanode.TestIncrementalBlockReports
            hadoop.hdfs.web.TestWebHdfsTokens
            hadoop.hdfs.TestSetrepIncreasing
            hadoop.fs.TestFcHdfsSetUMask
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.TestLargeBlock
            hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler
            hadoop.hdfs.TestFileAppend3
            hadoop.hdfs.server.blockmanagement.TestSequentialBlockId
            hadoop.hdfs.TestDFSClientFailover
            hadoop.hdfs.server.namenode.TestGetBlockLocations
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean
            hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.server.namenode.ha.TestHAMetrics
            hadoop.hdfs.web.TestJsonUtil
            hadoop.hdfs.server.namenode.TestFSDirectory
            hadoop.hdfs.server.namenode.TestEditLogAutoroll
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy
            hadoop.hdfs.server.namenode.TestAddBlockRetry
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestPipelines
            hadoop.hdfs.TestAppendDifferentChecksum
            hadoop.hdfs.server.namenode.TestTransferFsImage
            hadoop.hdfs.server.namenode.ha.TestHAFsck
            hadoop.hdfs.TestSeekBug
            hadoop.hdfs.TestFileAppend4
            hadoop.hdfs.TestDFSRemove
            hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
            hadoop.hdfs.TestBlockReaderLocalLegacy
            hadoop.hdfs.TestBlocksScheduledCounter
            hadoop.hdfs.TestExternalBlockReader
            hadoop.fs.viewfs.TestViewFsFileStatusHdfs
            hadoop.hdfs.TestSnapshotCommands
            hadoop.hdfs.server.datanode.TestDataNodeTransferSocketSize
            hadoop.fs.shell.TestHdfsTextCommand
            hadoop.hdfs.TestWriteConfigurationToDFS
            hadoop.hdfs.TestDFSRollback
            hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot
            hadoop.hdfs.TestFSInputChecker
            hadoop.hdfs.TestParallelShortCircuitRead
            hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
            hadoop.fs.TestGlobPaths
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
            hadoop.hdfs.TestHDFSTrash
            hadoop.hdfs.server.namenode.TestNameNodeAcl
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.fs.contract.hdfs.TestHDFSContractRename
            hadoop.hdfs.TestHDFSServerPorts
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
            hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
            hadoop.hdfs.TestHDFSFileSystemContract
            hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
            hadoop.net.TestNetworkTopology
            hadoop.hdfs.security.token.block.TestBlockToken
            hadoop.hdfs.server.datanode.TestIncrementalBrVariations
            hadoop.hdfs.server.datanode.TestHSync
            hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
            hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
            hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.namenode.TestCreateEditsLog
            hadoop.hdfs.server.namenode.TestFileContextXAttr
          Timed out tests org.apache.hadoop.hdfs.web.TestWebHDFS



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762175/HDFS-8873.009.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 71a81b6
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12661/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12661/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12661/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12661/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12661/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12661/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 48s There were no new javac warning messages. +1 javadoc 10m 27s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 8 new checkstyle issues (total was 439, now 439). -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 36s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 49m 46s Tests failed in hadoop-hdfs.     95m 33s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestFileStatus   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.namenode.TestINodeFile   hadoop.fs.contract.hdfs.TestHDFSContractOpen   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart   hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.fs.contract.hdfs.TestHDFSContractMkdir   hadoop.hdfs.TestDFSShell   hadoop.hdfs.server.namenode.TestNameNodeXAttr   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDecommission   hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.cli.TestXAttrCLI   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.server.namenode.TestParallelImageWrite   hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys   hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional   hadoop.hdfs.server.namenode.TestSaveNamespace   hadoop.hdfs.TestDFSRename   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.server.datanode.TestDeleteBlockPool   hadoop.hdfs.TestRemoteBlockReader2   hadoop.hdfs.server.namenode.TestStorageRestore   hadoop.hdfs.server.namenode.TestFileLimit   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestEncryptionZones   hadoop.fs.contract.hdfs.TestHDFSContractSetTimes   hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.namenode.TestFileContextAcl   hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestDataTransferProtocol   hadoop.fs.viewfs.TestViewFsWithXAttrs   hadoop.hdfs.server.datanode.TestDiskError   hadoop.hdfs.TestFsShellPermission   hadoop.hdfs.server.namenode.TestMalformedURLs   hadoop.hdfs.TestBalancerBandwidth   hadoop.hdfs.TestReadWhileWriting   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.hdfs.TestDisableConnCache   hadoop.hdfs.TestIsMethodSupported   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.server.blockmanagement.TestAvailableSpaceBlockPlacementPolicy   hadoop.hdfs.TestFileCreationClient   hadoop.cli.TestAclCLI   hadoop.hdfs.TestFSOutputSummer   hadoop.fs.TestHDFSFileContextMainOperations   hadoop.hdfs.TestDFSStartupVersions   hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir   hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure   hadoop.hdfs.server.namenode.TestGenericJournalConf   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.datanode.TestTransferRbw   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestFSImageWithAcl   hadoop.fs.TestUnbuffer   hadoop.hdfs.TestDFSPermission   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.hdfs.TestSmallBlock   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.tracing.TestTracing   hadoop.hdfs.server.datanode.TestTriggerBlockReport   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.fs.contract.hdfs.TestHDFSContractConcat   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.TestEditLog   hadoop.security.TestRefreshUserMappings   hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.server.namenode.TestHDFSConcat   hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.namenode.TestFSImageWithXAttr   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.TestDFSUtil   hadoop.security.TestPermissionSymlinks   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.fs.TestSymlinkHdfsDisable   hadoop.hdfs.server.namenode.TestQuotaByStorageType   hadoop.hdfs.TestFileCreation   hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.ha.TestQuotasWithHA   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits   hadoop.hdfs.server.namenode.ha.TestHAStateTransitions   hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot   hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps   hadoop.hdfs.TestRollingUpgradeDowngrade   hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover   hadoop.hdfs.TestBlockReaderFactory   hadoop.hdfs.TestDFSOutputStream   hadoop.fs.TestUrlStreamHandler   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.hdfs.TestModTime   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.fs.viewfs.TestViewFileSystemWithXAttrs   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.tools.TestGetGroups   hadoop.hdfs.server.namenode.TestNameNodeRecovery   hadoop.hdfs.TestRestartDFS   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.TestInjectionForSimulatedStorage   hadoop.hdfs.TestMiniDFSCluster   hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestLease   hadoop.hdfs.server.namenode.TestProtectedDirectories   hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot   hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.TestAuditLogs   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.fs.TestWebHdfsFileContextMainOperations   hadoop.hdfs.server.namenode.ha.TestNNHealthCheck   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.TestDFSAddressConfig   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics   hadoop.tools.TestJMXGet   hadoop.hdfs.server.namenode.TestEditLogJournalFailures   hadoop.hdfs.TestDFSMkdirs   hadoop.hdfs.TestWriteRead   hadoop.hdfs.TestClientBlockVerification   hadoop.fs.viewfs.TestViewFsDefaultValue   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.TestDatanodeConfig   hadoop.hdfs.server.namenode.TestNameNodeResourceChecker   hadoop.hdfs.server.datanode.TestDataNodeECN   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.namenode.TestAclConfigFlag   hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd   hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.TestAllowFormat   hadoop.hdfs.server.namenode.TestValidateConfigurationSettings   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.server.namenode.TestNameNodeRpcServer   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.fs.contract.hdfs.TestHDFSContractDelete   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter   hadoop.hdfs.TestDatanodeReport   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.TestAuditLogger   hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.TestDFSShellGenericOptions   hadoop.security.TestPermission   hadoop.hdfs.web.TestWebHDFSAcl   hadoop.hdfs.TestMissingBlocksAlert   hadoop.fs.viewfs.TestViewFsHdfs   hadoop.hdfs.TestClose   hadoop.cli.TestDeleteCLI   hadoop.hdfs.tools.TestDebugAdmin   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes   hadoop.fs.TestFcHdfsCreateMkdir   hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.namenode.TestXAttrConfigFlag   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.fs.permission.TestStickyBit   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.fs.viewfs.TestViewFsWithAcls   hadoop.hdfs.tools.TestStoragePolicyCommands   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.TestGenericRefresh   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestQuota   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   hadoop.fs.viewfs.TestViewFsAtHdfsRoot   hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.tracing.TestTracingShortCircuitLocalRead   hadoop.fs.TestFcHdfsPermission   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory   hadoop.fs.contract.hdfs.TestHDFSContractCreate   hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.web.TestWebHDFSOAuth2   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.fs.TestResolveHdfsSymlink   hadoop.hdfs.server.datanode.TestStorageReport   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.tracing.TestTraceAdmin   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename   hadoop.hdfs.server.namenode.TestNameEditsConfigs   hadoop.cli.TestCacheAdminCLI   hadoop.hdfs.server.datanode.TestDataNodeExit   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication   hadoop.hdfs.server.datanode.TestRefreshNamenodes   hadoop.hdfs.web.TestWebHDFSForHA   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.server.namenode.TestHostsFiles   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.TestEditLogRace   hadoop.hdfs.server.datanode.web.webhdfs.TestParameterParser   hadoop.TestRefreshCallQueue   hadoop.hdfs.TestSetTimes   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica   hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.fs.TestEnhancedByteBufferAccess   hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.hdfs.TestPersistBlocks   hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus   hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy   hadoop.hdfs.TestListFilesInFileContext   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestBlockMissingException   hadoop.hdfs.server.namenode.TestSnapshotPathINodes   hadoop.hdfs.server.datanode.web.dtp.TestDtpHttp2   hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade   hadoop.fs.viewfs.TestViewFileSystemHdfs   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.tools.TestDFSHAAdminMiniCluster   hadoop.hdfs.security.TestDelegationToken   hadoop.hdfs.server.namenode.TestBlockUnderConstruction   hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.TestINodeAttributeProvider   hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestRead   hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.hdfs.server.datanode.TestDataNodeInitStorage   hadoop.hdfs.server.namenode.TestFSNamesystemMBean   hadoop.fs.loadGenerator.TestLoadGenerator   hadoop.fs.contract.hdfs.TestHDFSContractAppend   hadoop.hdfs.server.namenode.TestSecondaryWebUi   hadoop.fs.viewfs.TestViewFileSystemWithAcls   hadoop.hdfs.web.TestWebHDFSXAttr   hadoop.hdfs.server.datanode.TestIncrementalBlockReports   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.TestSetrepIncreasing   hadoop.fs.TestFcHdfsSetUMask   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.TestLargeBlock   hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.server.blockmanagement.TestSequentialBlockId   hadoop.hdfs.TestDFSClientFailover   hadoop.hdfs.server.namenode.TestGetBlockLocations   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.server.namenode.ha.TestHAMetrics   hadoop.hdfs.web.TestJsonUtil   hadoop.hdfs.server.namenode.TestFSDirectory   hadoop.hdfs.server.namenode.TestEditLogAutoroll   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.hdfs.server.namenode.TestTransferFsImage   hadoop.hdfs.server.namenode.ha.TestHAFsck   hadoop.hdfs.TestSeekBug   hadoop.hdfs.TestFileAppend4   hadoop.hdfs.TestDFSRemove   hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.TestBlockReaderLocalLegacy   hadoop.hdfs.TestBlocksScheduledCounter   hadoop.hdfs.TestExternalBlockReader   hadoop.fs.viewfs.TestViewFsFileStatusHdfs   hadoop.hdfs.TestSnapshotCommands   hadoop.hdfs.server.datanode.TestDataNodeTransferSocketSize   hadoop.fs.shell.TestHdfsTextCommand   hadoop.hdfs.TestWriteConfigurationToDFS   hadoop.hdfs.TestDFSRollback   hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot   hadoop.hdfs.TestFSInputChecker   hadoop.hdfs.TestParallelShortCircuitRead   hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement   hadoop.fs.TestGlobPaths   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.server.namenode.TestNameNodeAcl   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.fs.contract.hdfs.TestHDFSContractRename   hadoop.hdfs.TestHDFSServerPorts   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr   hadoop.net.TestNetworkTopology   hadoop.hdfs.security.token.block.TestBlockToken   hadoop.hdfs.server.datanode.TestIncrementalBrVariations   hadoop.hdfs.server.datanode.TestHSync   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.TestCreateEditsLog   hadoop.hdfs.server.namenode.TestFileContextXAttr Timed out tests org.apache.hadoop.hdfs.web.TestWebHDFS Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762175/HDFS-8873.009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71a81b6 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12661/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12661/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12661/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12661/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12661/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12661/console This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Thanks Daniel Templeton for the update! I'm +1 (non-binding) for v9 of the patch.

          Show
          nroberts Nathan Roberts added a comment - Thanks Daniel Templeton for the update! I'm +1 (non-binding) for v9 of the patch.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Another repeat of the jenkins "noclassdef found" glitch. Re-triggering the build. +1 pending jenkins

          Show
          cmccabe Colin P. McCabe added a comment - Another repeat of the jenkins "noclassdef found" glitch. Re-triggering the build. +1 pending jenkins
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 1s There were no new javac warning messages.
          +1 javadoc 10m 15s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 26s The applied patch generated 8 new checkstyle issues (total was 439, now 439).
          -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 13s Pre-build of native portion
          -1 hdfs tests 162m 38s Tests failed in hadoop-hdfs.
              208m 35s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.TestDFSClientFailover
            hadoop.hdfs.web.TestWebHDFS
            hadoop.hdfs.TestDistributedFileSystem



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762175/HDFS-8873.009.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 83e99d0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12687/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12687/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12687/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12687/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12687/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12687/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 26s The applied patch generated 8 new checkstyle issues (total was 439, now 439). -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 13s Pre-build of native portion -1 hdfs tests 162m 38s Tests failed in hadoop-hdfs.     208m 35s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestDFSClientFailover   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.TestDistributedFileSystem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762175/HDFS-8873.009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e99d0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12687/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12687/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12687/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12687/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12687/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12687/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1. Thanks, Daniel Templeton. The test failures are unrelated (just more noclassfound jenkins environment issues), and all pass for me locally. Committed to 2.8 and trunk

          Show
          cmccabe Colin P. McCabe added a comment - +1. Thanks, Daniel Templeton . The test failures are unrelated (just more noclassfound jenkins environment issues), and all pass for me locally. Committed to 2.8 and trunk
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8523 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8523/)
          HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8523 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8523/ ) HDFS-8873 . Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #450 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/450/)
          HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #450 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/450/ ) HDFS-8873 . Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #444 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/444/)
          HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #444 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/444/ ) HDFS-8873 . Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1183 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1183/)
          HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1183 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1183/ ) HDFS-8873 . Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2388 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2388/)
          HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2388 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2388/ ) HDFS-8873 . Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2361 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2361/)
          HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2361 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2361/ ) HDFS-8873 . Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #421 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/421/)
          HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #421 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/421/ ) HDFS-8873 . Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) (cmccabe: rev 7a3c381b39887a02e944fa98287afd0eb4db3560) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          Hide
          wheat9 Haohui Mai added a comment -

          It looks like that this patch introduces a new findbugs warning in trunk. Please see

          https://builds.apache.org/job/PreCommit-HDFS-Build/12742/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html

          Show
          wheat9 Haohui Mai added a comment - It looks like that this patch introduces a new findbugs warning in trunk. Please see https://builds.apache.org/job/PreCommit-HDFS-Build/12742/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          Hide
          templedf Daniel Templeton added a comment -

          Already patched in HDFS-9174

          Show
          templedf Daniel Templeton added a comment - Already patched in HDFS-9174

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              nroberts Nathan Roberts
            • Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development