Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9782

RollingFileSystemSink should have configurable roll interval

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Right now it defaults to rolling at the top of every hour. Instead that interval should be configurable. The interval should also allow for some play so that all hosts don't try to flush their files simultaneously.

      I'm filing this in HDFS because I suspect it will involve touching the HDFS tests. If it turns out not to, I'll move it into common instead.

      1. HDFS-9782.009.patch
        57 kB
        Daniel Templeton
      2. HDFS-9782.008.patch
        56 kB
        Daniel Templeton
      3. HDFS-9782.007.patch
        52 kB
        Daniel Templeton
      4. HDFS-9782.006.patch
        53 kB
        Daniel Templeton
      5. HDFS-9782.005.patch
        50 kB
        Daniel Templeton
      6. HDFS-9782.004.patch
        47 kB
        Daniel Templeton
      7. HDFS-9782.003.patch
        47 kB
        Daniel Templeton
      8. HDFS-9782.002.patch
        39 kB
        Daniel Templeton
      9. HDFS-9782.001.patch
        8 kB
        Daniel Templeton

        Issue Links

          Activity

          Hide
          kasha Karthik Kambatla added a comment -

          Oh, and thanks Andrew Wang and Robert Kanter for your reviews.

          Show
          kasha Karthik Kambatla added a comment - Oh, and thanks Andrew Wang and Robert Kanter for your reviews.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for the contribution, Daniel Templeton. Just committed this to trunk and branch-2.

          Show
          kasha Karthik Kambatla added a comment - Thanks for the contribution, Daniel Templeton . Just committed this to trunk and branch-2.
          Hide
          kasha Karthik Kambatla added a comment -

          Okay, will check this into branch-2 then.

          Show
          kasha Karthik Kambatla added a comment - Okay, will check this into branch-2 then.
          Hide
          templedf Daniel Templeton added a comment -

          There's already a JIRA for it: HDFS-10449

          Show
          templedf Daniel Templeton added a comment - There's already a JIRA for it: HDFS-10449
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9847 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9847/)
          HDFS-9782. RollingFileSystemSink should have configurable roll interval. (kasha: rev 57c31a3fef625f1ec609d7e8873d4941f7ed5cbc)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithSecureHdfs.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithHdfs.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithLocal.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSink.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9847 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9847/ ) HDFS-9782 . RollingFileSystemSink should have configurable roll interval. (kasha: rev 57c31a3fef625f1ec609d7e8873d4941f7ed5cbc) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithSecureHdfs.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithHdfs.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithLocal.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSink.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java
          Hide
          kasha Karthik Kambatla added a comment -

          Committed to trunk, but TestRollingFileSystemSinkWithHdfs#testFailedClose fails on branch-2. Daniel Templeton - can you look into the test failure?

          Show
          kasha Karthik Kambatla added a comment - Committed to trunk, but TestRollingFileSystemSinkWithHdfs#testFailedClose fails on branch-2. Daniel Templeton - can you look into the test failure?
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for the updates, Daniel Templeton. +1, checking this in.

          Show
          kasha Karthik Kambatla added a comment - Thanks for the updates, Daniel Templeton . +1, checking this in.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 6m 19s trunk passed
          +1 compile 6m 20s trunk passed
          +1 checkstyle 1m 20s trunk passed
          +1 mvnsite 1m 47s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 2m 58s trunk passed
          +1 javadoc 2m 2s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 23s the patch passed
          +1 compile 6m 19s the patch passed
          +1 javac 6m 19s the patch passed
          -1 checkstyle 1m 19s root: patch generated 1 new + 12 unchanged - 6 fixed = 13 total (was 18)
          +1 mvnsite 1m 43s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 20s the patch passed
          +1 javadoc 2m 0s the patch passed
          -1 unit 19m 18s hadoop-common in the patch failed.
          -1 unit 59m 23s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          118m 21s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger
            hadoop.hdfs.TestCrcCorruption
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805875/HDFS-9782.009.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8c3674ce809f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b4078bd
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15545/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15545/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 6m 19s trunk passed +1 compile 6m 20s trunk passed +1 checkstyle 1m 20s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 58s trunk passed +1 javadoc 2m 2s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 6m 19s the patch passed +1 javac 6m 19s the patch passed -1 checkstyle 1m 19s root: patch generated 1 new + 12 unchanged - 6 fixed = 13 total (was 18) +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 20s the patch passed +1 javadoc 2m 0s the patch passed -1 unit 19m 18s hadoop-common in the patch failed. -1 unit 59m 23s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 118m 21s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger   hadoop.hdfs.TestCrcCorruption Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805875/HDFS-9782.009.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8c3674ce809f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b4078bd Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15545/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15545/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15545/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 40s Maven dependency ordering for branch
          +1 mvninstall 6m 46s trunk passed
          +1 compile 7m 42s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 2m 19s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 52s trunk passed
          +1 javadoc 2m 20s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 48s the patch passed
          +1 compile 8m 8s the patch passed
          +1 javac 8m 8s the patch passed
          -1 checkstyle 1m 31s root: patch generated 1 new + 12 unchanged - 6 fixed = 13 total (was 18)
          +1 mvnsite 2m 13s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 28s the patch passed
          +1 javadoc 2m 25s the patch passed
          +1 unit 8m 41s hadoop-common in the patch passed.
          -1 unit 64m 43s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 35s Patch does not generate ASF License warnings.
          122m 20s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestAsyncDFSRename



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805753/HDFS-9782.009.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b2675931efd0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6d043aa
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15525/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15525/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15525/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15525/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15525/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 40s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 7m 42s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 2m 19s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 52s trunk passed +1 javadoc 2m 20s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 48s the patch passed +1 compile 8m 8s the patch passed +1 javac 8m 8s the patch passed -1 checkstyle 1m 31s root: patch generated 1 new + 12 unchanged - 6 fixed = 13 total (was 18) +1 mvnsite 2m 13s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 28s the patch passed +1 javadoc 2m 25s the patch passed +1 unit 8m 41s hadoop-common in the patch passed. -1 unit 64m 43s hadoop-hdfs in the patch failed. +1 asflicense 0m 35s Patch does not generate ASF License warnings. 122m 20s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestAsyncDFSRename Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805753/HDFS-9782.009.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b2675931efd0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6d043aa Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15525/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15525/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15525/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15525/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15525/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          This should address the checkstyle issues.

          Show
          templedf Daniel Templeton added a comment - This should address the checkstyle issues.
          Hide
          kasha Karthik Kambatla added a comment -

          The v8 patch looks good to me, but for the checkstyle nit - couple of lines are longer than 80 chars.

          +1, outside of that. Will commit this soon after that is fixed.

          Show
          kasha Karthik Kambatla added a comment - The v8 patch looks good to me, but for the checkstyle nit - couple of lines are longer than 80 chars. +1, outside of that. Will commit this soon after that is fixed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 40s trunk passed
          +1 compile 6m 19s trunk passed
          +1 checkstyle 1m 17s trunk passed
          +1 mvnsite 1m 44s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 2m 59s trunk passed
          +1 javadoc 2m 4s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 6m 12s the patch passed
          +1 javac 6m 12s the patch passed
          -1 checkstyle 1m 20s root: patch generated 4 new + 14 unchanged - 5 fixed = 18 total (was 19)
          +1 mvnsite 1m 50s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 23s the patch passed
          +1 javadoc 2m 9s the patch passed
          +1 unit 9m 6s hadoop-common in the patch passed.
          -1 unit 79m 35s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          129m 3s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestAsyncDFSRename
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804826/HDFS-9782.008.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5394c4f787cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1597630
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15490/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15490/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15490/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15490/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15490/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 40s trunk passed +1 compile 6m 19s trunk passed +1 checkstyle 1m 17s trunk passed +1 mvnsite 1m 44s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 59s trunk passed +1 javadoc 2m 4s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 6m 12s the patch passed +1 javac 6m 12s the patch passed -1 checkstyle 1m 20s root: patch generated 4 new + 14 unchanged - 5 fixed = 18 total (was 19) +1 mvnsite 1m 50s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 23s the patch passed +1 javadoc 2m 9s the patch passed +1 unit 9m 6s hadoop-common in the patch passed. -1 unit 79m 35s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 129m 3s Reason Tests Failed junit tests hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestAsyncDFSRename Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804826/HDFS-9782.008.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5394c4f787cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15490/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15490/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15490/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15490/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15490/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 7m 11s trunk passed
          +1 compile 6m 18s trunk passed
          +1 checkstyle 1m 19s trunk passed
          +1 mvnsite 1m 44s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 11s trunk passed
          +1 javadoc 1m 56s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 33s the patch passed
          +1 compile 6m 22s the patch passed
          +1 javac 6m 22s the patch passed
          -1 checkstyle 1m 19s root: patch generated 4 new + 14 unchanged - 5 fixed = 18 total (was 19)
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 15s the patch passed
          +1 javadoc 2m 1s the patch passed
          -1 unit 6m 39s hadoop-common in the patch failed.
          -1 unit 58m 18s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          105m 29s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.hdfs.server.datanode.TestFsDatasetCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804826/HDFS-9782.008.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a81ddede2745 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 010e6ac
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15489/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15489/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 7m 11s trunk passed +1 compile 6m 18s trunk passed +1 checkstyle 1m 19s trunk passed +1 mvnsite 1m 44s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 11s trunk passed +1 javadoc 1m 56s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 33s the patch passed +1 compile 6m 22s the patch passed +1 javac 6m 22s the patch passed -1 checkstyle 1m 19s root: patch generated 4 new + 14 unchanged - 5 fixed = 18 total (was 19) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 15s the patch passed +1 javadoc 2m 1s the patch passed -1 unit 6m 39s hadoop-common in the patch failed. -1 unit 58m 18s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 105m 29s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.server.datanode.TestFsDatasetCache Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804826/HDFS-9782.008.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a81ddede2745 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 010e6ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15489/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15489/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15489/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the review, Karthik Kambatla!

          Is the empty constructor so Reflection works?

          Yep. I added JavaDocs to clarify that.

          The rest of your points should be addressed by the latest patch.

          Show
          templedf Daniel Templeton added a comment - Thanks for the review, Karthik Kambatla ! Is the empty constructor so Reflection works? Yep. I added JavaDocs to clarify that. The rest of your points should be addressed by the latest patch.
          Hide
          kasha Karthik Kambatla added a comment -

          Canceling patch to address comments.

          Show
          kasha Karthik Kambatla added a comment - Canceling patch to address comments.
          Hide
          kasha Karthik Kambatla added a comment -

          Sorry for the delay in getting to this. Looks mostly good. Some comments:

          1. Is the empty constructor so Reflection works?
          2. Javadoc for stringifySecurityProperty, findCurrentDirectory, createOrAppendLogFile, doTestGetRollInterval are broken. Mind fixing them?
          3. Nit: Should checkForProperty be renamed to checkIfPropertyExists for more clarity?
          4. RollingFileSystemSink#setInitialFlushTime is quite confusing to me. Can we clarify all the funkiness going on there? May be more comments? May be more meaningful variable name than millis?
          Show
          kasha Karthik Kambatla added a comment - Sorry for the delay in getting to this. Looks mostly good. Some comments: Is the empty constructor so Reflection works? Javadoc for stringifySecurityProperty, findCurrentDirectory, createOrAppendLogFile, doTestGetRollInterval are broken. Mind fixing them? Nit: Should checkForProperty be renamed to checkIfPropertyExists for more clarity? RollingFileSystemSink#setInitialFlushTime is quite confusing to me. Can we clarify all the funkiness going on there? May be more comments? May be more meaningful variable name than millis?
          Hide
          templedf Daniel Templeton added a comment -

          Test failures are all unrelated.

          Show
          templedf Daniel Templeton added a comment - Test failures are all unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 6m 26s trunk passed
          +1 compile 5m 37s trunk passed with JDK v1.8.0_77
          +1 compile 6m 31s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 2s trunk passed
          +1 mvnsite 1m 46s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 27s trunk passed
          +1 javadoc 1m 53s trunk passed with JDK v1.8.0_77
          +1 javadoc 2m 51s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 5m 34s the patch passed with JDK v1.8.0_77
          +1 javac 5m 34s the patch passed
          +1 compile 6m 32s the patch passed with JDK v1.7.0_95
          +1 javac 6m 32s the patch passed
          +1 checkstyle 1m 2s the patch passed
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 56s the patch passed
          +1 javadoc 1m 54s the patch passed with JDK v1.8.0_77
          +1 javadoc 2m 45s the patch passed with JDK v1.7.0_95
          -1 unit 21m 2s hadoop-common in the patch failed with JDK v1.8.0_77.
          -1 unit 58m 5s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 6m 51s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 54m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          198m 21s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.net.TestDNS
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestCrcCorruption
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_95 Failed junit tests hadoop.net.TestDNS
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797308/HDFS-9782.007.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5fd5c84b330f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b41e65e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15092/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15092/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 6m 26s trunk passed +1 compile 5m 37s trunk passed with JDK v1.8.0_77 +1 compile 6m 31s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 2s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 27s trunk passed +1 javadoc 1m 53s trunk passed with JDK v1.8.0_77 +1 javadoc 2m 51s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 5m 34s the patch passed with JDK v1.8.0_77 +1 javac 5m 34s the patch passed +1 compile 6m 32s the patch passed with JDK v1.7.0_95 +1 javac 6m 32s the patch passed +1 checkstyle 1m 2s the patch passed +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 56s the patch passed +1 javadoc 1m 54s the patch passed with JDK v1.8.0_77 +1 javadoc 2m 45s the patch passed with JDK v1.7.0_95 -1 unit 21m 2s hadoop-common in the patch failed with JDK v1.8.0_77. -1 unit 58m 5s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 6m 51s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 54m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 198m 21s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestCrcCorruption JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_95 Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797308/HDFS-9782.007.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5fd5c84b330f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b41e65e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15092/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15092/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15092/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          This patch removes seconds an a valid unit for the roll interval.

          Show
          templedf Daniel Templeton added a comment - This patch removes seconds an a valid unit for the roll interval.
          Hide
          kasha Karthik Kambatla added a comment -

          I see your point, but given the unlikeliness of someone differentiating between 30 seconds and a 1 minute, I ll keep it simple and drop seconds altogether.

          Show
          kasha Karthik Kambatla added a comment - I see your point, but given the unlikeliness of someone differentiating between 30 seconds and a 1 minute, I ll keep it simple and drop seconds altogether.
          Hide
          templedf Daniel Templeton added a comment -

          The granularity of the directory names is 1 minute, but that doesn't require that the interval have a 1-minute granularity. I was thinking of the case of a 90-second interval. If you think that has the potential to create confusion, I can drop seconds from the allowed units.

          Show
          templedf Daniel Templeton added a comment - The granularity of the directory names is 1 minute, but that doesn't require that the interval have a 1-minute granularity. I was thinking of the case of a 90-second interval. If you think that has the potential to create confusion, I can drop seconds from the allowed units.
          Hide
          kasha Karthik Kambatla added a comment -

          Quickly skimmed through the patch. One major comment: if we want to support only minute granularity for this, why allow users to specify seconds? The offset milliseconds sounds okay, because the purpose is different.

          Show
          kasha Karthik Kambatla added a comment - Quickly skimmed through the patch. One major comment: if we want to support only minute granularity for this, why allow users to specify seconds? The offset milliseconds sounds okay, because the purpose is different.
          Hide
          templedf Daniel Templeton added a comment -

          I just finished some manual testing, and everything appears to be working as intended. This thing's ready for review.

          Show
          templedf Daniel Templeton added a comment - I just finished some manual testing, and everything appears to be working as intended. This thing's ready for review.
          Hide
          templedf Daniel Templeton added a comment -

          Test failures are still unrelated. These seem a little less suspicious. 2 out of 3 clean, with 2 out of the last 2 clean is good enough for me.

          Show
          templedf Daniel Templeton added a comment - Test failures are still unrelated. These seem a little less suspicious. 2 out of 3 clean, with 2 out of the last 2 clean is good enough for me.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 1m 43s Maven dependency ordering for branch
          +1 mvninstall 9m 21s trunk passed
          +1 compile 10m 29s trunk passed with JDK v1.8.0_72
          +1 compile 8m 0s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 21s trunk passed
          +1 mvnsite 2m 50s trunk passed
          +1 mvneclipse 0m 47s trunk passed
          +1 findbugs 4m 15s trunk passed
          +1 javadoc 3m 2s trunk passed with JDK v1.8.0_72
          +1 javadoc 4m 31s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 21s Maven dependency ordering for patch
          +1 mvninstall 1m 45s the patch passed
          +1 compile 9m 5s the patch passed with JDK v1.8.0_72
          +1 javac 9m 5s the patch passed
          +1 compile 9m 47s the patch passed with JDK v1.7.0_95
          +1 javac 9m 47s the patch passed
          +1 checkstyle 1m 15s the patch passed
          +1 mvnsite 2m 37s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 29s the patch passed
          +1 javadoc 2m 24s the patch passed with JDK v1.8.0_72
          +1 javadoc 3m 7s the patch passed with JDK v1.7.0_95
          -1 unit 10m 11s hadoop-common in the patch failed with JDK v1.8.0_72.
          -1 unit 96m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 11m 26s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 89m 9s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 40s Patch does not generate ASF License warnings.
          292m 55s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.fs.shell.find.TestPrint
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790365/HDFS-9782.006.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ccd18357b2fb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7545ce6
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14643/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14643/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 1m 43s Maven dependency ordering for branch +1 mvninstall 9m 21s trunk passed +1 compile 10m 29s trunk passed with JDK v1.8.0_72 +1 compile 8m 0s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 21s trunk passed +1 mvnsite 2m 50s trunk passed +1 mvneclipse 0m 47s trunk passed +1 findbugs 4m 15s trunk passed +1 javadoc 3m 2s trunk passed with JDK v1.8.0_72 +1 javadoc 4m 31s trunk passed with JDK v1.7.0_95 0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 1m 45s the patch passed +1 compile 9m 5s the patch passed with JDK v1.8.0_72 +1 javac 9m 5s the patch passed +1 compile 9m 47s the patch passed with JDK v1.7.0_95 +1 javac 9m 47s the patch passed +1 checkstyle 1m 15s the patch passed +1 mvnsite 2m 37s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 29s the patch passed +1 javadoc 2m 24s the patch passed with JDK v1.8.0_72 +1 javadoc 3m 7s the patch passed with JDK v1.7.0_95 -1 unit 10m 11s hadoop-common in the patch failed with JDK v1.8.0_72. -1 unit 96m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 11m 26s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 89m 9s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 40s Patch does not generate ASF License warnings. 292m 55s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.fs.shell.find.TestPrint   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790365/HDFS-9782.006.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ccd18357b2fb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7545ce6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14643/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14643/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14643/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Since we have some time to kill, bumping Jenkins again to see if I can get a set of unit test results that don't look suspicious.

          Show
          templedf Daniel Templeton added a comment - Since we have some time to kill, bumping Jenkins again to see if I can get a set of unit test results that don't look suspicious.
          Hide
          templedf Daniel Templeton added a comment -

          Test failures still look a little suspicious, but none of them are related this time.

          Show
          templedf Daniel Templeton added a comment - Test failures still look a little suspicious, but none of them are related this time.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 36s trunk passed
          +1 compile 8m 38s trunk passed with JDK v1.8.0_72
          +1 compile 7m 28s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 2m 6s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 35s trunk passed
          +1 javadoc 2m 24s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 11s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 37s the patch passed
          +1 compile 8m 30s the patch passed with JDK v1.8.0_72
          +1 javac 8m 30s the patch passed
          +1 compile 7m 47s the patch passed with JDK v1.7.0_95
          +1 javac 7m 47s the patch passed
          +1 checkstyle 1m 4s the patch passed
          +1 mvnsite 2m 10s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 11s the patch passed
          +1 javadoc 2m 20s the patch passed with JDK v1.8.0_72
          +1 javadoc 3m 11s the patch passed with JDK v1.7.0_95
          -1 unit 9m 41s hadoop-common in the patch failed with JDK v1.8.0_72.
          -1 unit 81m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 9m 47s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 78m 44s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          250m 22s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.namenode.TestFileTruncate



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790320/HDFS-9782.006.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fd14ab5969e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 03cfb45
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14641/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14641/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 36s trunk passed +1 compile 8m 38s trunk passed with JDK v1.8.0_72 +1 compile 7m 28s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 2m 6s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 35s trunk passed +1 javadoc 2m 24s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 11s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 37s the patch passed +1 compile 8m 30s the patch passed with JDK v1.8.0_72 +1 javac 8m 30s the patch passed +1 compile 7m 47s the patch passed with JDK v1.7.0_95 +1 javac 7m 47s the patch passed +1 checkstyle 1m 4s the patch passed +1 mvnsite 2m 10s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 11s the patch passed +1 javadoc 2m 20s the patch passed with JDK v1.8.0_72 +1 javadoc 3m 11s the patch passed with JDK v1.7.0_95 -1 unit 9m 41s hadoop-common in the patch failed with JDK v1.8.0_72. -1 unit 81m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 9m 47s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 78m 44s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 250m 22s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790320/HDFS-9782.006.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fd14ab5969e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 03cfb45 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14641/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14641/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          The test failures look suspicious. Bumping Jenkins to have another go at it.

          Show
          templedf Daniel Templeton added a comment - The test failures look suspicious. Bumping Jenkins to have another go at it.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 7m 13s trunk passed
          +1 compile 8m 19s trunk passed with JDK v1.8.0_72
          +1 compile 7m 21s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 59s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 35s trunk passed
          +1 javadoc 2m 20s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 3s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 30s the patch passed
          +1 compile 8m 12s the patch passed with JDK v1.8.0_72
          +1 javac 8m 12s the patch passed
          +1 compile 7m 20s the patch passed with JDK v1.7.0_95
          +1 javac 7m 20s the patch passed
          +1 checkstyle 1m 5s the patch passed
          +1 mvnsite 1m 58s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 0s the patch passed
          +1 javadoc 2m 17s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 59s the patch passed with JDK v1.7.0_95
          +1 unit 9m 47s hadoop-common in the patch passed with JDK v1.8.0_72.
          -1 unit 94m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 11m 56s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 81m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 40s Patch does not generate ASF License warnings.
          266m 28s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestFileAppend
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          JDK v1.7.0_95 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
            hadoop.hdfs.TestRollingUpgrade
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790315/HDFS-9782.006.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 84c2f9d24472 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 798babf
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14640/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14640/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 13s trunk passed +1 compile 8m 19s trunk passed with JDK v1.8.0_72 +1 compile 7m 21s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 59s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 35s trunk passed +1 javadoc 2m 20s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 3s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 8m 12s the patch passed with JDK v1.8.0_72 +1 javac 8m 12s the patch passed +1 compile 7m 20s the patch passed with JDK v1.7.0_95 +1 javac 7m 20s the patch passed +1 checkstyle 1m 5s the patch passed +1 mvnsite 1m 58s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 0s the patch passed +1 javadoc 2m 17s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 59s the patch passed with JDK v1.7.0_95 +1 unit 9m 47s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 94m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 11m 56s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 81m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 40s Patch does not generate ASF License warnings. 266m 28s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestFileAppend   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport JDK v1.7.0_95 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.TestRollingUpgrade   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790315/HDFS-9782.006.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 84c2f9d24472 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 798babf Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14640/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14640/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Resolved checkstyle issue.

          Show
          templedf Daniel Templeton added a comment - Resolved checkstyle issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 42s Maven dependency ordering for branch
          +1 mvninstall 7m 6s trunk passed
          +1 compile 6m 39s trunk passed with JDK v1.8.0_72
          +1 compile 7m 22s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 2m 4s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 46s trunk passed
          +1 javadoc 2m 7s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 59s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 34s the patch passed
          +1 compile 6m 43s the patch passed with JDK v1.8.0_72
          +1 javac 6m 43s the patch passed
          +1 compile 7m 19s the patch passed with JDK v1.7.0_95
          +1 javac 7m 19s the patch passed
          -1 checkstyle 1m 8s root: patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 mvnsite 2m 1s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 12s the patch passed
          +1 javadoc 2m 6s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 59s the patch passed with JDK v1.7.0_95
          +1 unit 8m 32s hadoop-common in the patch passed with JDK v1.8.0_72.
          -1 unit 58m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 8m 36s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 56m 9s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          208m 28s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790284/HDFS-9782.005.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 64ef6df7a47f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0f72da7
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14638/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14638/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14638/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14638/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14638/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 42s Maven dependency ordering for branch +1 mvninstall 7m 6s trunk passed +1 compile 6m 39s trunk passed with JDK v1.8.0_72 +1 compile 7m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 9s trunk passed +1 mvnsite 2m 4s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 46s trunk passed +1 javadoc 2m 7s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 59s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 6m 43s the patch passed with JDK v1.8.0_72 +1 javac 6m 43s the patch passed +1 compile 7m 19s the patch passed with JDK v1.7.0_95 +1 javac 7m 19s the patch passed -1 checkstyle 1m 8s root: patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 mvnsite 2m 1s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 12s the patch passed +1 javadoc 2m 6s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 59s the patch passed with JDK v1.7.0_95 +1 unit 8m 32s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 58m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 8m 36s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 56m 9s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 208m 28s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790284/HDFS-9782.005.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 64ef6df7a47f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0f72da7 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14638/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14638/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14638/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14638/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14638/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Turns out that HADOOP-8608 doesn't actually help me. It adds the getTimeDuration() method to Configuration, but metrics are initialized with a SubsetConfiguration (from Apache Commons). Looks like I still have to do the parsing by hand. I have switched over to using TimeUnit for the conversions, though.

          After poking at trying to fake the clock, I came to a useful realization. The BPServiceActor tests are just trying to test the timing. I'm already doing that in the TestRollingFileSystemSink tests. What I'm trying to test in the test with the sleeps is whether the flush thread successfully flushes the logs, which can only be tested by actually scheduling it to run. With that in mind, I found a way to test that functionality with no sleeps in the common case. The sleeps are still there, just in case, but I've never seen it sleep even once. I also bumped the max sleep time in the test way up so that the chance of flakiness is approximately 0.

          I still need to do more manual testing, but first let's see if this passes muster.

          Show
          templedf Daniel Templeton added a comment - Turns out that HADOOP-8608 doesn't actually help me. It adds the getTimeDuration() method to Configuration , but metrics are initialized with a SubsetConfiguration (from Apache Commons). Looks like I still have to do the parsing by hand. I have switched over to using TimeUnit for the conversions, though. After poking at trying to fake the clock, I came to a useful realization. The BPServiceActor tests are just trying to test the timing. I'm already doing that in the TestRollingFileSystemSink tests. What I'm trying to test in the test with the sleeps is whether the flush thread successfully flushes the logs, which can only be tested by actually scheduling it to run. With that in mind, I found a way to test that functionality with no sleeps in the common case. The sleeps are still there, just in case, but I've never seen it sleep even once. I also bumped the max sleep time in the test way up so that the chance of flakiness is approximately 0. I still need to do more manual testing, but first let's see if this passes muster.
          Hide
          rkanter Robert Kanter added a comment -

          The unit parsing stuff isn't a blocker for me either. Hopefully that will all get consolidated eventually. Though I agree with Andrew Wang's earlier comment to use TimeUnit.convert.

          A 30 second default sounds good to me.

          Show
          rkanter Robert Kanter added a comment - The unit parsing stuff isn't a blocker for me either. Hopefully that will all get consolidated eventually. Though I agree with Andrew Wang 's earlier comment to use TimeUnit.convert. A 30 second default sounds good to me.
          Hide
          templedf Daniel Templeton added a comment -

          I'm OK with 30 seconds as a default.

          Show
          templedf Daniel Templeton added a comment - I'm OK with 30 seconds as a default.
          Hide
          andrew.wang Andrew Wang added a comment -

          Based on numbers I've seen, the NN can do a few hundred files per second, so throwing a couple hundred or thousand at the NN all at once will result in a multi-second blip. A little fuzz goes a long way here, so if you're cool with 1min or even 30s, I think that's sufficient. Speaking from experience, even big cluster operators aren't necessarily more savvy about Hadoop config keys.

          There is also a meta point about timeliness. There's always going to be inaccuracy in data collection (NTP fail, GC pause, dog chewed an Ethernet cable), and this needs to be accounted for when processing. This is like the famous "lambda architecture" from the streaming world; handle late data in a rollup.

          Show
          andrew.wang Andrew Wang added a comment - Based on numbers I've seen, the NN can do a few hundred files per second, so throwing a couple hundred or thousand at the NN all at once will result in a multi-second blip. A little fuzz goes a long way here, so if you're cool with 1min or even 30s, I think that's sufficient. Speaking from experience, even big cluster operators aren't necessarily more savvy about Hadoop config keys. There is also a meta point about timeliness. There's always going to be inaccuracy in data collection (NTP fail, GC pause, dog chewed an Ethernet cable), and this needs to be accounted for when processing. This is like the famous "lambda architecture" from the streaming world; handle late data in a rollup.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks, Andrew Wang! HADOOP-8608 appears to be exactly what we want.

          Fair point about needing to deal with a GC pause, but having the offset on by default still strikes me as a potentially nasty surprise. I'm trying to think about this in terms of customer experience. We know that there's no noticeable performance impact at 200 nodes. We're just assuming that we'll run into issues at larger scale, but we don't actually know for sure. It just seems wrong to me to add this little bit of unexpected uncertainty into the mix for all users when we suspect that a handful of users might run into the issue. Also consider that an admin running a 1000-node cluster is going to be a bit more careful when changing configuration settings than someone with a 10-node cluster. The big cluster's admin is less likely to be surprised by needing to turn on the offset than the little cluster's admin will be about it being on by default.

          The above begs the question about what the default should be. If we think it's 1 second or even 10 seconds, I'll stop arguing now and turn it on by default. I assumed we'd want something more like 1 minute. At a minute, that's long enough that some user will trip over it and be confused. I don't think more than 1 minute is a reasonable default for several reasons, one of which is that it could interact badly with a short roll interval. (I don't think it makes sense to set the offset as a percentage of the roll interval, because the need for the offset is independent of the length of the roll interval.)

          What kind of timeliness do we really require? Would it be acceptable if we did not synchronize rolling, but rolled more frequently?

          The use case behind this JIRA requires log rolls at the top of every hour with a known time by which the logs are guaranteed to be available. Having a 1 minute offset as the default is fine for this use case. The discussion we're having here is about all the use cases we haven't seen yet. Sorry to suck up time on such an trivial detail, but I think it's worth getting right.

          Show
          templedf Daniel Templeton added a comment - Thanks, Andrew Wang ! HADOOP-8608 appears to be exactly what we want. Fair point about needing to deal with a GC pause, but having the offset on by default still strikes me as a potentially nasty surprise. I'm trying to think about this in terms of customer experience. We know that there's no noticeable performance impact at 200 nodes. We're just assuming that we'll run into issues at larger scale, but we don't actually know for sure. It just seems wrong to me to add this little bit of unexpected uncertainty into the mix for all users when we suspect that a handful of users might run into the issue. Also consider that an admin running a 1000-node cluster is going to be a bit more careful when changing configuration settings than someone with a 10-node cluster. The big cluster's admin is less likely to be surprised by needing to turn on the offset than the little cluster's admin will be about it being on by default. The above begs the question about what the default should be. If we think it's 1 second or even 10 seconds, I'll stop arguing now and turn it on by default. I assumed we'd want something more like 1 minute. At a minute, that's long enough that some user will trip over it and be confused. I don't think more than 1 minute is a reasonable default for several reasons, one of which is that it could interact badly with a short roll interval. (I don't think it makes sense to set the offset as a percentage of the roll interval, because the need for the offset is independent of the length of the roll interval.) What kind of timeliness do we really require? Would it be acceptable if we did not synchronize rolling, but rolled more frequently? The use case behind this JIRA requires log rolls at the top of every hour with a known time by which the logs are guaranteed to be available. Having a 1 minute offset as the default is fine for this use case. The discussion we're having here is about all the use cases we haven't seen yet. Sorry to suck up time on such an trivial detail, but I think it's worth getting right.
          Hide
          andrew.wang Andrew Wang added a comment -

          My concern is that the offset interval alters when the metrics are reliably available. I think it violates the principal of least astonishment to have the metrics randomly (literally) show up late by default. I would rather it not be on unless it's needed, and the user turns it on explicitly.

          Is it that weird? You just need to poll offset after the flush. You also always need to be able to deal with late data, since the flush could pause or be delayed for other reasons too (e.g. GC pause).

          I'm still not entirely clear on the requirements, since I can't think of other windowed metrics that we try to synchronize cluster wide. What kind of timeliness do we really require? Would it be acceptable if we did not synchronize rolling, but rolled more frequently?

          What's the alternative? I don't think millis is an acceptable unit for something that will likely be hours or days.

          I did some JIRA searching, and found HADOOP-8608 which I didn't realize was available. Is this what we want?

          Show
          andrew.wang Andrew Wang added a comment - My concern is that the offset interval alters when the metrics are reliably available. I think it violates the principal of least astonishment to have the metrics randomly (literally) show up late by default. I would rather it not be on unless it's needed, and the user turns it on explicitly. Is it that weird? You just need to poll offset after the flush. You also always need to be able to deal with late data, since the flush could pause or be delayed for other reasons too (e.g. GC pause). I'm still not entirely clear on the requirements, since I can't think of other windowed metrics that we try to synchronize cluster wide. What kind of timeliness do we really require? Would it be acceptable if we did not synchronize rolling, but rolled more frequently? What's the alternative? I don't think millis is an acceptable unit for something that will likely be hours or days. I did some JIRA searching, and found HADOOP-8608 which I didn't realize was available. Is this what we want?
          Hide
          templedf Daniel Templeton added a comment -

          Wow, checkstyle really doesn't like case statements to be indented...

          Thanks for jumping in, Andrew Wang!

          If your concern is the linking between the interval and the offset, we could make the offset configuration a percent of the interval.

          My concern is that the offset interval alters when the metrics are reliably available. I think it violates the principal of least astonishment to have the metrics randomly (literally) show up late by default. I would rather it not be on unless it's needed, and the user turns it on explicitly.

          I also agree with Robert and would prefer that we didn't add this unit parsing code at all, but that's not a blocker.

          What's the alternative? I don't think millis is an acceptable unit for something that will likely be hours or days.

          Also, if you look at BPServiceActor#Scheduler, this is an example of how we can unit test a scheduler like this without sleeps. Food for thought.

          Now I get what you meant in HDFS-9637 about testing using a clock that can be set by the tests. That seems pretty reasonable. I clearly need to get better acquainted with Mockito.

          I'll take another pass at it.

          Show
          templedf Daniel Templeton added a comment - Wow, checkstyle really doesn't like case statements to be indented... Thanks for jumping in, Andrew Wang ! If your concern is the linking between the interval and the offset, we could make the offset configuration a percent of the interval. My concern is that the offset interval alters when the metrics are reliably available. I think it violates the principal of least astonishment to have the metrics randomly (literally) show up late by default. I would rather it not be on unless it's needed, and the user turns it on explicitly. I also agree with Robert and would prefer that we didn't add this unit parsing code at all, but that's not a blocker. What's the alternative? I don't think millis is an acceptable unit for something that will likely be hours or days. Also, if you look at BPServiceActor#Scheduler, this is an example of how we can unit test a scheduler like this without sleeps. Food for thought. Now I get what you meant in HDFS-9637 about testing using a clock that can be set by the tests. That seems pretty reasonable. I clearly need to get better acquainted with Mockito. I'll take another pass at it.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 1m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 32s trunk passed
          +1 compile 5m 36s trunk passed with JDK v1.8.0_72
          +1 compile 6m 30s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 52s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 24s trunk passed
          +1 javadoc 2m 1s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 49s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 6m 22s the patch passed with JDK v1.8.0_72
          +1 javac 6m 22s the patch passed
          +1 compile 6m 43s the patch passed with JDK v1.7.0_95
          +1 javac 6m 43s the patch passed
          -1 checkstyle 1m 3s root: patch generated 25 new + 0 unchanged - 0 fixed = 25 total (was 0)
          +1 mvnsite 1m 52s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 56s the patch passed
          +1 javadoc 2m 9s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 56s the patch passed with JDK v1.7.0_95
          +1 unit 6m 49s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 55m 1s hadoop-hdfs in the patch passed with JDK v1.8.0_72.
          -1 unit 6m 38s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 51m 11s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          180m 12s



          Reason Tests
          JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789618/HDFS-9782.004.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 449c7640bb7d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d27d7fc
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14600/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14600/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14600/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14600/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14600/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 1m 9s Maven dependency ordering for branch +1 mvninstall 6m 32s trunk passed +1 compile 5m 36s trunk passed with JDK v1.8.0_72 +1 compile 6m 30s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 24s trunk passed +1 javadoc 2m 1s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 49s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 6m 22s the patch passed with JDK v1.8.0_72 +1 javac 6m 22s the patch passed +1 compile 6m 43s the patch passed with JDK v1.7.0_95 +1 javac 6m 43s the patch passed -1 checkstyle 1m 3s root: patch generated 25 new + 0 unchanged - 0 fixed = 25 total (was 0) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 56s the patch passed +1 javadoc 2m 9s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 56s the patch passed with JDK v1.7.0_95 +1 unit 6m 49s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 55m 1s hadoop-hdfs in the patch passed with JDK v1.8.0_72. -1 unit 6m 38s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 51m 11s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 180m 12s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789618/HDFS-9782.004.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 449c7640bb7d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d27d7fc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14600/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14600/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14600/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14600/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14600/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          In most clusters, this is not needed. It's only the large (1000-ish node) clusters that will need to worry about staggering the rolls. And then how much staggering is required depends heavily on the cluster. I think 0 is a reasonable default.

          Is there a downside to having it non-zero for small clusters? It's better to have defaults that work for all cluster sizes. If your concern is the linking between the interval and the offset, we could make the offset configuration a percent of the interval.

          One nit, we can use TimeUnit.convert rather than using the new constants. I also agree with Robert and would prefer that we didn't add this unit parsing code at all, but that's not a blocker.

          Also, if you look at BPServiceActor#Scheduler, this is an example of how we can unit test a scheduler like this without sleeps. Food for thought.

          Show
          andrew.wang Andrew Wang added a comment - In most clusters, this is not needed. It's only the large (1000-ish node) clusters that will need to worry about staggering the rolls. And then how much staggering is required depends heavily on the cluster. I think 0 is a reasonable default. Is there a downside to having it non-zero for small clusters? It's better to have defaults that work for all cluster sizes. If your concern is the linking between the interval and the offset, we could make the offset configuration a percent of the interval. One nit, we can use TimeUnit.convert rather than using the new constants. I also agree with Robert and would prefer that we didn't add this unit parsing code at all, but that's not a blocker. Also, if you look at BPServiceActor#Scheduler, this is an example of how we can unit test a scheduler like this without sleeps. Food for thought.
          Hide
          templedf Daniel Templeton added a comment -

          Robert Kanter, here's a patch fixing the typo. I'm holding off on other changes until we see eye to eye.

          Show
          templedf Daniel Templeton added a comment - Robert Kanter , here's a patch fixing the typo. I'm holding off on other changes until we see eye to eye.
          Hide
          templedf Daniel Templeton added a comment -

          Robert Kanter, thank you for the review!

          If the idea here is to prevent attacking HDFS with everyone rolling at the same time, I think the default value should not be 0. That basically negates the what we're trying to do here.

          In most clusters, this is not needed. It's only the large (1000-ish node) clusters that will need to worry about staggering the rolls. And then how much staggering is required depends heavily on the cluster. I think 0 is a reasonable default.

          I'm not sure we should try to conform to HDFS-9821 here at this point.

          Perhaps I overstated things a little. I was already allowing for user-specified units when HDFS-9821 was created. I liked the way they proposed to do it better, so I changed my code to work that way instead. I agree that at some point there may be some shared utils to parse the time, but I need to do it now regardless.

          And I'm not worried about roll-offset-interval-millis. I think that one should actually stay only in millis.

          On a slow system or with some other delay, this could easily cause the test to be flakey...

          I see your point, but it would have to be an enormously overloaded system. The thread will run at the top of the second, so it's scheduled to run in less than 1000ms. If it takes more than 500ms to do the flush, things are seriously FUBAR. All it's doing is closing a file. I kinda think it should fail at that point.

          Show
          templedf Daniel Templeton added a comment - Robert Kanter , thank you for the review! If the idea here is to prevent attacking HDFS with everyone rolling at the same time, I think the default value should not be 0. That basically negates the what we're trying to do here. In most clusters, this is not needed. It's only the large (1000-ish node) clusters that will need to worry about staggering the rolls. And then how much staggering is required depends heavily on the cluster. I think 0 is a reasonable default. I'm not sure we should try to conform to HDFS-9821 here at this point. Perhaps I overstated things a little. I was already allowing for user-specified units when HDFS-9821 was created. I liked the way they proposed to do it better, so I changed my code to work that way instead. I agree that at some point there may be some shared utils to parse the time, but I need to do it now regardless. And I'm not worried about roll-offset-interval-millis . I think that one should actually stay only in millis. On a slow system or with some other delay, this could easily cause the test to be flakey... I see your point, but it would have to be an enormously overloaded system. The thread will run at the top of the second, so it's scheduled to run in less than 1000ms. If it takes more than 500ms to do the flush, things are seriously FUBAR. All it's doing is closing a file. I kinda think it should fail at that point.
          Hide
          rkanter Robert Kanter added a comment -

          Looks good overall. A few things:

          • Typo: "...some variance is the roll times..."
          • If the idea here is to prevent attacking HDFS with everyone rolling at the same time, I think the default value should not be 0. That basically negates the what we're trying to do here.
          • I'm not sure we should try to conform to HDFS-9821 here at this point. You have to define a lot of extra code to handle the parsing. I imagine HDFS-9821 will eventually make a common place for these and take care of it transparently to the code that's actually using the config property's value; we don't want a bunch of different implementations of this. Instead, it sounds like you should be able to make this "compatible" by naming the key roll-offset-interval instead of roll-offset-interval-millis.
          • On a slow system or with some other delay, this could easily cause the test to be flakey:
                int count = 0;
            
                // Sleep until the flusher has run
                while (!RollingFileSystemSink.hasFlushed) {
                  Thread.sleep(100L);
            
                  if (++count > 15) {
                    fail("Flush thread did not run within 1.5 seconds");
                  }
                }
            
          Show
          rkanter Robert Kanter added a comment - Looks good overall. A few things: Typo: "...some variance is the roll times..." If the idea here is to prevent attacking HDFS with everyone rolling at the same time, I think the default value should not be 0. That basically negates the what we're trying to do here. I'm not sure we should try to conform to HDFS-9821 here at this point. You have to define a lot of extra code to handle the parsing. I imagine HDFS-9821 will eventually make a common place for these and take care of it transparently to the code that's actually using the config property's value; we don't want a bunch of different implementations of this. Instead, it sounds like you should be able to make this "compatible" by naming the key roll-offset-interval instead of roll-offset-interval-millis . On a slow system or with some other delay, this could easily cause the test to be flakey: int count = 0; // Sleep until the flusher has run while (!RollingFileSystemSink.hasFlushed) { Thread .sleep(100L); if (++count > 15) { fail( "Flush thread did not run within 1.5 seconds" ); } }
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 42s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 47s Maven dependency ordering for branch
          +1 mvninstall 8m 8s trunk passed
          +1 compile 9m 47s trunk passed with JDK v1.8.0_72
          +1 compile 8m 21s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 2m 28s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          +1 findbugs 3m 56s trunk passed
          +1 javadoc 2m 25s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 14s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 38s the patch passed
          +1 compile 8m 52s the patch passed with JDK v1.8.0_72
          +1 javac 8m 52s the patch passed
          +1 compile 8m 15s the patch passed with JDK v1.7.0_95
          +1 javac 8m 15s the patch passed
          -1 checkstyle 1m 24s root: patch generated 25 new + 0 unchanged - 0 fixed = 25 total (was 0)
          +1 mvnsite 2m 31s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 1s the patch passed
          +1 javadoc 3m 25s the patch passed with JDK v1.8.0_72
          +1 javadoc 4m 6s the patch passed with JDK v1.7.0_95
          -1 unit 11m 23s hadoop-common in the patch failed with JDK v1.8.0_72.
          -1 unit 92m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 9m 0s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 73m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 29s Patch does not generate ASF License warnings.
          266m 14s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestRollingUpgrade
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788917/HDFS-9782.003.patch
          JIRA Issue HDFS-9782
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f22f36fa2520 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d5abd29
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14554/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14554/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 42s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 47s Maven dependency ordering for branch +1 mvninstall 8m 8s trunk passed +1 compile 9m 47s trunk passed with JDK v1.8.0_72 +1 compile 8m 21s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 9s trunk passed +1 mvnsite 2m 28s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 3m 56s trunk passed +1 javadoc 2m 25s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 14s trunk passed with JDK v1.7.0_95 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 38s the patch passed +1 compile 8m 52s the patch passed with JDK v1.8.0_72 +1 javac 8m 52s the patch passed +1 compile 8m 15s the patch passed with JDK v1.7.0_95 +1 javac 8m 15s the patch passed -1 checkstyle 1m 24s root: patch generated 25 new + 0 unchanged - 0 fixed = 25 total (was 0) +1 mvnsite 2m 31s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 1s the patch passed +1 javadoc 3m 25s the patch passed with JDK v1.8.0_72 +1 javadoc 4m 6s the patch passed with JDK v1.7.0_95 -1 unit 11m 23s hadoop-common in the patch failed with JDK v1.8.0_72. -1 unit 92m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 9m 0s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 73m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 29s Patch does not generate ASF License warnings. 266m 14s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestRollingUpgrade   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788917/HDFS-9782.003.patch JIRA Issue HDFS-9782 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f22f36fa2520 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d5abd29 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14554/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14554/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14554/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Oops. I meant to conform with HDFS-9821.

          Show
          templedf Daniel Templeton added a comment - Oops. I meant to conform with HDFS-9821 .
          Hide
          templedf Daniel Templeton added a comment -

          This patch changes the roll interval property to conform with HDFS-9829. It also cleans up the secure HDFS test (from HDFS-9780) to run faster.

          Show
          templedf Daniel Templeton added a comment - This patch changes the roll interval property to conform with HDFS-9829 . It also cleans up the secure HDFS test (from HDFS-9780 ) to run faster.
          Hide
          templedf Daniel Templeton added a comment -

          And here's a complete patch. I moved the previous TestRollingFileSystemSink to TestRollingFileSystemSinkWithLocal and created a new TestRollingFileSystemSink for proper unit tests.

          Show
          templedf Daniel Templeton added a comment - And here's a complete patch. I moved the previous TestRollingFileSystemSink to TestRollingFileSystemSinkWithLocal and created a new TestRollingFileSystemSink for proper unit tests.
          Hide
          templedf Daniel Templeton added a comment -

          Here's an initial sketch of the plan.

          Show
          templedf Daniel Templeton added a comment - Here's an initial sketch of the plan.
          Hide
          templedf Daniel Templeton added a comment -

          In HDFS-9780, Andrew Wang suggested the interval be milliseconds. Given that most intervals are going to be on the order of hours, an interval of milliseconds seems cruel. How many milliseconds in a day? Plus any interval less than about 10 minutes is at risk of creating a problematic number of small files. For these reasons I'm going to ignore Andrew's suggestion and go with minutes as the interval.

          Show
          templedf Daniel Templeton added a comment - In HDFS-9780 , Andrew Wang suggested the interval be milliseconds. Given that most intervals are going to be on the order of hours, an interval of milliseconds seems cruel. How many milliseconds in a day? Plus any interval less than about 10 minutes is at risk of creating a problematic number of small files. For these reasons I'm going to ignore Andrew's suggestion and go with minutes as the interval.

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              templedf Daniel Templeton
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development