Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12759

RollingFileSystemSink should eagerly rotate directories

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      The RollingFileSystemSink only rolls over to a new directory if a new metrics record comes in. The issue is that HDFS does not update the file size until it's closed (HDFS-5478), and if no new metrics record comes in, then the file size will never be updated.

      This JIRA is to add a background thread to the sink that will eagerly close the file at the top of the hour.

      1. HADOOP-12759.003.patch
        20 kB
        Daniel Templeton
      2. HADOOP-12759.002.patch
        20 kB
        Daniel Templeton
      3. YARN-4664.001.patch
        13 kB
        Daniel Templeton

        Issue Links

          Activity

          Hide
          templedf Daniel Templeton added a comment -

          The additional test for the new flushing behavior will go into HDFS-9637.

          Show
          templedf Daniel Templeton added a comment - The additional test for the new flushing behavior will go into HDFS-9637 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 7m 5s trunk passed
          +1 compile 7m 2s trunk passed with JDK v1.8.0_66
          +1 compile 7m 30s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 42s the patch passed
          +1 compile 6m 39s the patch passed with JDK v1.8.0_66
          +1 javac 6m 39s the patch passed
          +1 compile 7m 23s the patch passed with JDK v1.7.0_91
          +1 javac 7m 23s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 10s the patch passed
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 5s the patch passed with JDK v1.7.0_91
          +1 unit 8m 54s hadoop-common in the patch passed with JDK v1.8.0_66.
          -1 unit 8m 36s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          67m 34s



          Reason Tests
          JDK v1.7.0_91 Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785773/YARN-4664.001.patch
          JIRA Issue HADOOP-12759
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1a21393f297b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 700a176
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8511/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8511/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8511/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8511/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 5s trunk passed +1 compile 7m 2s trunk passed with JDK v1.8.0_66 +1 compile 7m 30s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 42s the patch passed +1 compile 6m 39s the patch passed with JDK v1.8.0_66 +1 javac 6m 39s the patch passed +1 compile 7m 23s the patch passed with JDK v1.7.0_91 +1 javac 7m 23s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 10s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 5s the patch passed with JDK v1.7.0_91 +1 unit 8m 54s hadoop-common in the patch passed with JDK v1.8.0_66. -1 unit 8m 36s hadoop-common in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 67m 34s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785773/YARN-4664.001.patch JIRA Issue HADOOP-12759 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1a21393f297b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 700a176 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8511/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8511/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8511/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8511/console This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          A couple things:

          1. I'm not a big fan of adding special configs for tests. Any ideas on a different way to handle this? I'm not really sure.
          2. IIRC, removing the throws IOException from the close() method wouldn't be backwards compatible (a caller could fail at compile time if they were expected an IOException to be thrown and it no longer is).
          Show
          rkanter Robert Kanter added a comment - A couple things: I'm not a big fan of adding special configs for tests. Any ideas on a different way to handle this? I'm not really sure. IIRC, removing the throws IOException from the close() method wouldn't be backwards compatible (a caller could fail at compile time if they were expected an IOException to be thrown and it no longer is).
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Daniel, I'm coming into this fresh, so please excuse my comments as I get up to speed on this. Overall looks good, only nitty stuff, then some questions:

          • Not a fan of a test config key since it's exposed to end users, can we use a static variable or a VisibleForTesting setter instead? I didn't see any related tests in HDFS-9637. I'm hoping whatever test emerges does not involve Thread.sleep, since I hate sleeping in unit tests.
          • The probing logic, instead of trying creates until we find a free file, should we list the directory once first? Or once after the first failed create, then probe?
          • Need <p/> tags to get line breaks in class javadoc.

          Some high-level or commentary or nits:

          • In the penultimate paragraph of the class javadoc, do you know why reads fail? I'd believe close failing if the pipeline strength falls (HDFS-4504), but reads failing after a successful close is surprising. This is generally only an issue with small clusters.
          • An aside comment, since HDFS always writes one block to the local DN, it can lead to skew if there's only one or few writers. Just an FYI depending on your usecase.
          Show
          andrew.wang Andrew Wang added a comment - Hi Daniel, I'm coming into this fresh, so please excuse my comments as I get up to speed on this. Overall looks good, only nitty stuff, then some questions: Not a fan of a test config key since it's exposed to end users, can we use a static variable or a VisibleForTesting setter instead? I didn't see any related tests in HDFS-9637 . I'm hoping whatever test emerges does not involve Thread.sleep, since I hate sleeping in unit tests. The probing logic, instead of trying creates until we find a free file, should we list the directory once first? Or once after the first failed create, then probe? Need <p/> tags to get line breaks in class javadoc. Some high-level or commentary or nits: In the penultimate paragraph of the class javadoc, do you know why reads fail? I'd believe close failing if the pipeline strength falls ( HDFS-4504 ), but reads failing after a successful close is surprising. This is generally only an issue with small clusters. An aside comment, since HDFS always writes one block to the local DN, it can lead to skew if there's only one or few writers. Just an FYI depending on your usecase.
          Hide
          andrew.wang Andrew Wang added a comment -

          I did not refresh before posting my comment, good to see that Robert and I have the same feelings about test-only configs

          Show
          andrew.wang Andrew Wang added a comment - I did not refresh before posting my comment, good to see that Robert and I have the same feelings about test-only configs
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the reviews, Robert Kanter and Andrew Wang!

          I'll give it a go with a static variable and add some paragraph tags.

          On the probing logic, the reason I do it that way is to get synchronization across daemons. I let HDFS sort out who gets any given file name. If I list files first, the list of files could change by the time I go to create the file.

          On removing the throws clause, I'm pretty sure that has no adverse effect. It still adhere's to the interface regardless of whether the throws cause is contracted or the return type is more specific.

          About sleeping in the flush test, let's talk alternatives. I did actually write the test with a sleep. If I don't wait for the flusher thread to roll over and flush, then how do I test it? I can force the flush to happen on demand, but if I do that, I'm not testing the timing of the thread; I'm just testing that the flush behaves as expected. Do we just assume that timing works and only test the behavior?

          Show
          templedf Daniel Templeton added a comment - Thanks for the reviews, Robert Kanter and Andrew Wang ! I'll give it a go with a static variable and add some paragraph tags. On the probing logic, the reason I do it that way is to get synchronization across daemons. I let HDFS sort out who gets any given file name. If I list files first, the list of files could change by the time I go to create the file. On removing the throws clause, I'm pretty sure that has no adverse effect. It still adhere's to the interface regardless of whether the throws cause is contracted or the return type is more specific. About sleeping in the flush test, let's talk alternatives. I did actually write the test with a sleep. If I don't wait for the flusher thread to roll over and flush, then how do I test it? I can force the flush to happen on demand, but if I do that, I'm not testing the timing of the thread; I'm just testing that the flush behaves as expected. Do we just assume that timing works and only test the behavior?
          Hide
          templedf Daniel Templeton added a comment -

          Playing around with the test a bit, I can comfortably get the sleep down to about half a second. Less than that, and it gets unreliable. Any heartburn about a half-second sleep?

          Show
          templedf Daniel Templeton added a comment - Playing around with the test a bit, I can comfortably get the sleep down to about half a second. Less than that, and it gets unreliable. Any heartburn about a half-second sleep?
          Hide
          andrew.wang Andrew Wang added a comment -

          On the probing logic, the reason I do it that way is to get synchronization across daemons. I let HDFS sort out who gets any given file name. If I list files first, the list of files could change by the time I go to create the file.

          I mentioned this off-hand in my previous comment, but how about we try once, if it fails list to find the last element and try n+1, then keep probing linearly until it works. This is then no overhead for the common case (no collisions) and we skip to the end if there is a conflict. Intent is to avoid a full linear probe.

          Any heartburn about a half-second sleep?

          Tolerable heartburn, but I was hoping for some solution with advancing a fake clock and then waking up the sleeping thread. I'll still +1 though if you don't want to change this.

          Show
          andrew.wang Andrew Wang added a comment - On the probing logic, the reason I do it that way is to get synchronization across daemons. I let HDFS sort out who gets any given file name. If I list files first, the list of files could change by the time I go to create the file. I mentioned this off-hand in my previous comment, but how about we try once, if it fails list to find the last element and try n+1, then keep probing linearly until it works. This is then no overhead for the common case (no collisions) and we skip to the end if there is a conflict. Intent is to avoid a full linear probe. Any heartburn about a half-second sleep? Tolerable heartburn, but I was hoping for some solution with advancing a fake clock and then waking up the sleeping thread. I'll still +1 though if you don't want to change this.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 16m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 58s trunk passed
          +1 compile 6m 5s trunk passed with JDK v1.8.0_72
          +1 compile 6m 48s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 32s trunk passed
          +1 javadoc 0m 53s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 42s the patch passed
          +1 compile 6m 5s the patch passed with JDK v1.8.0_72
          +1 javac 6m 5s the patch passed
          +1 compile 6m 47s the patch passed with JDK v1.7.0_95
          +1 javac 6m 47s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 50s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 54s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 4s the patch passed with JDK v1.7.0_95
          +1 unit 7m 16s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 7m 32s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          76m 31s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Write to static field org.apache.hadoop.metrics2.sink.RollingFileSystemSink.hasFlushed from instance method org.apache.hadoop.metrics2.sink.RollingFileSystemSink.init(SubsetConfiguration) At RollingFileSystemSink.java:from instance method org.apache.hadoop.metrics2.sink.RollingFileSystemSink.init(SubsetConfiguration) At RollingFileSystemSink.java:[line 166]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786335/HADOOP-12759.002.patch
          JIRA Issue HADOOP-12759
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1e628390db30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1bcfab8
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8540/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8540/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8540/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 16m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 6m 58s trunk passed +1 compile 6m 5s trunk passed with JDK v1.8.0_72 +1 compile 6m 48s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 53s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 42s the patch passed +1 compile 6m 5s the patch passed with JDK v1.8.0_72 +1 javac 6m 5s the patch passed +1 compile 6m 47s the patch passed with JDK v1.7.0_95 +1 javac 6m 47s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 50s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 54s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 4s the patch passed with JDK v1.7.0_95 +1 unit 7m 16s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 7m 32s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 76m 31s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Write to static field org.apache.hadoop.metrics2.sink.RollingFileSystemSink.hasFlushed from instance method org.apache.hadoop.metrics2.sink.RollingFileSystemSink.init(SubsetConfiguration) At RollingFileSystemSink.java:from instance method org.apache.hadoop.metrics2.sink.RollingFileSystemSink.init(SubsetConfiguration) At RollingFileSystemSink.java: [line 166] Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786335/HADOOP-12759.002.patch JIRA Issue HADOOP-12759 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1e628390db30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1bcfab8 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8540/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8540/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8540/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          New patch to address findbugs warning.

          Show
          templedf Daniel Templeton added a comment - New patch to address findbugs warning.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 5m 43s trunk passed with JDK v1.8.0_72
          +1 compile 6m 31s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 32s trunk passed
          +1 javadoc 0m 52s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 39s the patch passed
          +1 compile 5m 37s the patch passed with JDK v1.8.0_72
          +1 javac 5m 37s the patch passed
          +1 compile 6m 33s the patch passed with JDK v1.7.0_95
          +1 javac 6m 33s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95
          +1 unit 7m 17s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 7m 22s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          59m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786672/HADOOP-12759.003.patch
          JIRA Issue HADOOP-12759
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 420d51d73901 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 29ae258
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8560/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8560/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 5m 43s trunk passed with JDK v1.8.0_72 +1 compile 6m 31s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 52s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 39s the patch passed +1 compile 5m 37s the patch passed with JDK v1.8.0_72 +1 javac 5m 37s the patch passed +1 compile 6m 33s the patch passed with JDK v1.7.0_95 +1 javac 6m 33s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95 +1 unit 7m 17s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 7m 22s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 59m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786672/HADOOP-12759.003.patch JIRA Issue HADOOP-12759 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 420d51d73901 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 29ae258 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8560/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8560/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          LGTM I'll commit shortly

          Show
          andrew.wang Andrew Wang added a comment - LGTM I'll commit shortly
          Hide
          andrew.wang Andrew Wang added a comment -

          Pushed to trunk and branch-2, the original patch HADOOP-12702 is not in branch-2.8 so I'm updating the target/fixed version to 2.9.0.

          Thanks for the contribution Daniel and review Robert!

          Show
          andrew.wang Andrew Wang added a comment - Pushed to trunk and branch-2, the original patch HADOOP-12702 is not in branch-2.8 so I'm updating the target/fixed version to 2.9.0. Thanks for the contribution Daniel and review Robert!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9256 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9256/)
          HADOOP-12759. RollingFileSystemSink should eagerly rotate directories. (wang: rev 5b59a0ea85c923384e36ad7c036e751551774142)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9256 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9256/ ) HADOOP-12759 . RollingFileSystemSink should eagerly rotate directories. (wang: rev 5b59a0ea85c923384e36ad7c036e751551774142) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              templedf Daniel Templeton
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development