Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9780

RollingFileSystemSink doesn't work on secure clusters

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If HDFS has kerberos enabled, the sink cannot write its logs.

      1. HADOOP-12775.001.patch
        8 kB
        Daniel Templeton
      2. HADOOP-12775.002.patch
        10 kB
        Daniel Templeton
      3. HADOOP-12775.003.patch
        11 kB
        Daniel Templeton
      4. HDFS-9780.004.patch
        21 kB
        Daniel Templeton
      5. HDFS-9780.005.patch
        25 kB
        Daniel Templeton
      6. HDFS-9780.006.patch
        25 kB
        Daniel Templeton
      7. HDFS-9780.006.patch
        25 kB
        Daniel Templeton
      8. HDFS-9780.007.patch
        26 kB
        Daniel Templeton
      9. HDFS-9780.008.patch
        26 kB
        Daniel Templeton

        Issue Links

          Activity

          Hide
          templedf Daniel Templeton added a comment -

          Hey, Karthik Kambatla, ping!

          Show
          templedf Daniel Templeton added a comment - Hey, Karthik Kambatla , ping!
          Hide
          templedf Daniel Templeton added a comment -

          Hey, Karthik Kambatla, this should be closed, right?

          Show
          templedf Daniel Templeton added a comment - Hey, Karthik Kambatla , this should be closed, right?
          Hide
          cnauroth Chris Nauroth added a comment -

          Daniel Templeton, thanks for taking a look.

          Show
          cnauroth Chris Nauroth added a comment - Daniel Templeton , thanks for taking a look.
          Hide
          templedf Daniel Templeton added a comment -

          The issue is not this patch. You're seeing HADOOP-12090 rearing its head. The message happened to be split across two TCP packets, causing the KDC to reject it. If you look at the stack trace in the HDFS-9711 logs, it's identical to HADOOP-12090:

          EXCEPTION
          org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 43 6C 82 02 3F 30 82 02 3B A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C2 30 82 01 BE 30 82 01 BA A1 03 02 01 01 A2 82 01 B1 04 82 01 AD 6E 82 01 A9 30 82 01 A5 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F5 61 81 F2 30 81 EF A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B6 30 81 B3 A0 03 02 01 11 A2 81 AB 04 81 A8 CD 9D D5 D9 84 E1 75 0B AE 35 44 84 E6 25 3D 07 9F 15 43 E8 31 D3 8D 2A 45 41 F2 2B B4 4D C4 0F 24 A2 70 F2 53 2A C7 A9 9C 93 54 43 6E C0 5F B4 47 49 7F 49 C1 96 18 4E F1 F6 07 55 A7 B2 2C 6C 6A D1 82 60 A1 04 29 D2 BB 38 6D D9 88 40 7B 85 24 CD B3 37 04 08 34 4D C4 24 EB 45 CE AD 70 6C 3A C2 8E 73 60 6F D4 36 44 D0 01 62 75 73 85 91 98 54 59 05 E8 B4 34 72 00 30 D5 4F 4C 39 4D D9 4C 5E 46 C6 63 DB 6D B0 15 D6 88 0A F0 C5 C8 43 A7 9D 71 69 AF 80 4A 52 DB 8D 0B E0 C0 80 24 28 EF DC F4 71 B5 2F 91 64 A4 81 97 30 81 94 A0 03 02 01 11 A2 81 8C 04 81 89 17 CE ED C3 54 CD A3 A0 81 CE 3C 31 B4 09 2C 29 A2 B4 30 5D 88 28 04 75 15 CA D7 3C 0C B2 DF 07 84 A4 0D AF 71 EB 55 39 D4 A1 C0 70 99 9A 11 9E DA C3 96 ED C2 6F 3E 32 F8 55 83 0C F8 3E 8E 05 20 46 49 02 AF 0C 6A 53 58 80 93 F4 14 86 A9 60 2E 00 CD BD 6E 93 23 9D 8B 5C D9 D8 96 23 56 AB 61 7F 78 29 29 98 86 33 48 A2 11 F7 6F 11 39 CE D4 FB F4 DE 76 88 70 A7 09 8C CA A8 BC C6 89 A6 7D 7C FB 50 EA 47 7B 43 7F A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 68 64 66 73 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 4E 5A 84 B4 A8 14 30 12 02 01 12 02 01 11 02 01 10 02 01)
          	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
          	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
          	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
          	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
          	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
          	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
          	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
          	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
          	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
          	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
          	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
          	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
          	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          	at java.lang.Thread.run(Thread.java:745)
          Caused by: java.lang.NullPointerException: message
          	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
          	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
          	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
          	... 15 more
          
          Show
          templedf Daniel Templeton added a comment - The issue is not this patch. You're seeing HADOOP-12090 rearing its head. The message happened to be split across two TCP packets, causing the KDC to reject it. If you look at the stack trace in the HDFS-9711 logs, it's identical to HADOOP-12090 : EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 43 6C 82 02 3F 30 82 02 3B A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C2 30 82 01 BE 30 82 01 BA A1 03 02 01 01 A2 82 01 B1 04 82 01 AD 6E 82 01 A9 30 82 01 A5 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F5 61 81 F2 30 81 EF A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B6 30 81 B3 A0 03 02 01 11 A2 81 AB 04 81 A8 CD 9D D5 D9 84 E1 75 0B AE 35 44 84 E6 25 3D 07 9F 15 43 E8 31 D3 8D 2A 45 41 F2 2B B4 4D C4 0F 24 A2 70 F2 53 2A C7 A9 9C 93 54 43 6E C0 5F B4 47 49 7F 49 C1 96 18 4E F1 F6 07 55 A7 B2 2C 6C 6A D1 82 60 A1 04 29 D2 BB 38 6D D9 88 40 7B 85 24 CD B3 37 04 08 34 4D C4 24 EB 45 CE AD 70 6C 3A C2 8E 73 60 6F D4 36 44 D0 01 62 75 73 85 91 98 54 59 05 E8 B4 34 72 00 30 D5 4F 4C 39 4D D9 4C 5E 46 C6 63 DB 6D B0 15 D6 88 0A F0 C5 C8 43 A7 9D 71 69 AF 80 4A 52 DB 8D 0B E0 C0 80 24 28 EF DC F4 71 B5 2F 91 64 A4 81 97 30 81 94 A0 03 02 01 11 A2 81 8C 04 81 89 17 CE ED C3 54 CD A3 A0 81 CE 3C 31 B4 09 2C 29 A2 B4 30 5D 88 28 04 75 15 CA D7 3C 0C B2 DF 07 84 A4 0D AF 71 EB 55 39 D4 A1 C0 70 99 9A 11 9E DA C3 96 ED C2 6F 3E 32 F8 55 83 0C F8 3E 8E 05 20 46 49 02 AF 0C 6A 53 58 80 93 F4 14 86 A9 60 2E 00 CD BD 6E 93 23 9D 8B 5C D9 D8 96 23 56 AB 61 7F 78 29 29 98 86 33 48 A2 11 F7 6F 11 39 CE D4 FB F4 DE 76 88 70 A7 09 8C CA A8 BC C6 89 A6 7D 7C FB 50 EA 47 7B 43 7F A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 68 64 66 73 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 4E 5A 84 B4 A8 14 30 12 02 01 12 02 01 11 02 01 10 02 01) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more
          Hide
          cnauroth Chris Nauroth added a comment -

          I am seeing TestRollingFileSystemSinkWithHdfs fail with Kerberos errors when running on JDK 7. It seems to pass fine on JDK 8. See HDFS-9711 for an example of it failing in pre-commit. Shall we revert this, or would one of the original contributors like to take a look first?

          Show
          cnauroth Chris Nauroth added a comment - I am seeing TestRollingFileSystemSinkWithHdfs fail with Kerberos errors when running on JDK 7. It seems to pass fine on JDK 8. See HDFS-9711 for an example of it failing in pre-commit. Shall we revert this, or would one of the original contributors like to take a look first?
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9288 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9288/)
          HDFS-9780. RollingFileSystemSink doesn't work on secure clusters. (kasha: rev 33ce7f6c072144f55be30c66099eef5bc736405e)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithSecureHdfs.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithHdfs.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9288 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9288/ ) HDFS-9780 . RollingFileSystemSink doesn't work on secure clusters. (kasha: rev 33ce7f6c072144f55be30c66099eef5bc736405e) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithSecureHdfs.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSinkWithHdfs.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          kasha Karthik Kambatla added a comment -

          Checking this in.

          Show
          kasha Karthik Kambatla added a comment - Checking this in.
          Hide
          templedf Daniel Templeton added a comment -

          The test failure is unrelated. Robert Kanter and Karthik Kambatla, wanna pull the trigger?

          Show
          templedf Daniel Templeton added a comment - The test failure is unrelated. Robert Kanter and Karthik Kambatla , wanna pull the trigger?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 21s trunk passed
          +1 compile 6m 41s trunk passed with JDK v1.8.0_66
          +1 compile 7m 18s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 8s trunk passed
          +1 mvnsite 2m 6s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 45s trunk passed
          +1 javadoc 2m 6s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 0s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 32s the patch passed
          +1 compile 6m 42s the patch passed with JDK v1.8.0_66
          +1 javac 6m 42s the patch passed
          +1 compile 7m 25s the patch passed with JDK v1.7.0_91
          +1 javac 7m 25s the patch passed
          +1 checkstyle 1m 6s the patch passed
          +1 mvnsite 2m 6s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 17s the patch passed
          +1 javadoc 2m 8s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 1s the patch passed with JDK v1.7.0_91
          +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.8.0_66.
          -1 unit 59m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 55m 21s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          196m 38s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787548/HDFS-9780.008.patch
          JIRA Issue HDFS-9780
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b764b959f162 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 23f937e
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14461/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14461/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14461/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 76MB
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14461/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 21s trunk passed +1 compile 6m 41s trunk passed with JDK v1.8.0_66 +1 compile 7m 18s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 8s trunk passed +1 mvnsite 2m 6s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 45s trunk passed +1 javadoc 2m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 0s trunk passed with JDK v1.7.0_91 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 32s the patch passed +1 compile 6m 42s the patch passed with JDK v1.8.0_66 +1 javac 6m 42s the patch passed +1 compile 7m 25s the patch passed with JDK v1.7.0_91 +1 javac 7m 25s the patch passed +1 checkstyle 1m 6s the patch passed +1 mvnsite 2m 6s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 17s the patch passed +1 javadoc 2m 8s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 1s the patch passed with JDK v1.7.0_91 +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.8.0_66. -1 unit 59m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 55m 21s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 196m 38s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787548/HDFS-9780.008.patch JIRA Issue HDFS-9780 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b764b959f162 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23f937e Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14461/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14461/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14461/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 76MB Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14461/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          +1 will commit once Jenkins says it's okay.

          Show
          rkanter Robert Kanter added a comment - +1 will commit once Jenkins says it's okay.
          Hide
          kasha Karthik Kambatla added a comment -

          +1 pending Jenkins.

          Show
          kasha Karthik Kambatla added a comment - +1 pending Jenkins.
          Hide
          templedf Daniel Templeton added a comment -

          Tiny update to deal with deprecation issue.

          Show
          templedf Daniel Templeton added a comment - Tiny update to deal with deprecation issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 5m 43s trunk passed with JDK v1.8.0_72
          +1 compile 6m 44s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 5s trunk passed
          +1 mvnsite 1m 56s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 29s trunk passed
          +1 javadoc 2m 1s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 50s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 5m 57s the patch passed with JDK v1.8.0_72
          -1 javac 7m 40s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 740 unchanged - 0 fixed = 741 total (was 740)
          +1 javac 5m 57s the patch passed
          +1 compile 6m 42s the patch passed with JDK v1.7.0_95
          -1 javac 14m 23s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 735 unchanged - 0 fixed = 736 total (was 735)
          +1 javac 6m 42s the patch passed
          +1 checkstyle 1m 4s the patch passed
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 53s the patch passed
          +1 javadoc 2m 2s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 51s the patch passed with JDK v1.7.0_95
          -1 unit 6m 56s hadoop-common in the patch failed with JDK v1.8.0_72.
          +1 unit 68m 24s hadoop-hdfs in the patch passed with JDK v1.8.0_72.
          +1 unit 7m 42s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 69m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 54s Patch does not generate ASF License warnings.
          213m 12s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787393/HDFS-9780.007.patch
          JIRA Issue HDFS-9780
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 155ef3dd4786 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aeb13ef
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          javac root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_72.txt
          javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14451/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 76MB
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14451/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 5m 43s trunk passed with JDK v1.8.0_72 +1 compile 6m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 5s trunk passed +1 mvnsite 1m 56s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 29s trunk passed +1 javadoc 2m 1s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 50s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 5m 57s the patch passed with JDK v1.8.0_72 -1 javac 7m 40s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 740 unchanged - 0 fixed = 741 total (was 740) +1 javac 5m 57s the patch passed +1 compile 6m 42s the patch passed with JDK v1.7.0_95 -1 javac 14m 23s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 735 unchanged - 0 fixed = 736 total (was 735) +1 javac 6m 42s the patch passed +1 checkstyle 1m 4s the patch passed +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 53s the patch passed +1 javadoc 2m 2s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 51s the patch passed with JDK v1.7.0_95 -1 unit 6m 56s hadoop-common in the patch failed with JDK v1.8.0_72. +1 unit 68m 24s hadoop-hdfs in the patch passed with JDK v1.8.0_72. +1 unit 7m 42s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 69m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 54s Patch does not generate ASF License warnings. 213m 12s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787393/HDFS-9780.007.patch JIRA Issue HDFS-9780 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 155ef3dd4786 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aeb13ef Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 javac root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_72.txt javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14451/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14451/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 76MB Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14451/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 45s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 41s Maven dependency ordering for branch
          +1 mvninstall 6m 33s trunk passed
          +1 compile 5m 33s trunk passed with JDK v1.8.0_72
          +1 compile 6m 27s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 54s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 20s trunk passed
          +1 javadoc 1m 55s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 48s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 5m 30s the patch passed with JDK v1.8.0_72
          -1 javac 7m 9s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 739 unchanged - 0 fixed = 740 total (was 739)
          +1 javac 5m 30s the patch passed
          +1 compile 6m 30s the patch passed with JDK v1.7.0_95
          -1 javac 13m 39s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 735 unchanged - 0 fixed = 736 total (was 735)
          +1 javac 6m 30s the patch passed
          +1 checkstyle 1m 2s the patch passed
          +1 mvnsite 1m 50s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 50s the patch passed
          +1 javadoc 1m 56s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 50s the patch passed with JDK v1.7.0_95
          +1 unit 6m 40s hadoop-common in the patch passed with JDK v1.8.0_72.
          -1 unit 52m 56s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 6m 32s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 50m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          189m 33s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestReconstructStripedFile
          JDK v1.7.0_95 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.server.namenode.TestCacheDirectives



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787393/HDFS-9780.007.patch
          JIRA Issue HDFS-9780
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9ae160717aa9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aeb13ef
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          javac root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_72.txt
          javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14450/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 76MB
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14450/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 45s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 41s Maven dependency ordering for branch +1 mvninstall 6m 33s trunk passed +1 compile 5m 33s trunk passed with JDK v1.8.0_72 +1 compile 6m 27s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 54s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 20s trunk passed +1 javadoc 1m 55s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 48s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 5m 30s the patch passed with JDK v1.8.0_72 -1 javac 7m 9s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 739 unchanged - 0 fixed = 740 total (was 739) +1 javac 5m 30s the patch passed +1 compile 6m 30s the patch passed with JDK v1.7.0_95 -1 javac 13m 39s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 735 unchanged - 0 fixed = 736 total (was 735) +1 javac 6m 30s the patch passed +1 checkstyle 1m 2s the patch passed +1 mvnsite 1m 50s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 50s the patch passed +1 javadoc 1m 56s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 50s the patch passed with JDK v1.7.0_95 +1 unit 6m 40s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 52m 56s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 6m 32s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 50m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 189m 33s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestReconstructStripedFile JDK v1.7.0_95 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.server.namenode.TestCacheDirectives Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787393/HDFS-9780.007.patch JIRA Issue HDFS-9780 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9ae160717aa9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aeb13ef Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 javac root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_72.txt javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14450/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14450/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 76MB Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14450/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Good suggestion, Karthik Kambatla! New patch posted.

          Show
          templedf Daniel Templeton added a comment - Good suggestion, Karthik Kambatla ! New patch posted.
          Hide
          kasha Karthik Kambatla added a comment -

          Patch is looking good, and thanks for updating the tests to not use the secure config by default.

          That said, the check in setup method to run it only for tests without secure in their name seems a little brittle - it is easy to miss and add a secure test without saying Secure. May be we could move them to another class that inherits from the same base?

          Show
          kasha Karthik Kambatla added a comment - Patch is looking good, and thanks for updating the tests to not use the secure config by default. That said, the check in setup method to run it only for tests without secure in their name seems a little brittle - it is easy to miss and add a secure test without saying Secure. May be we could move them to another class that inherits from the same base?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 10s Maven dependency ordering for branch
          +1 mvninstall 9m 23s trunk passed
          +1 compile 11m 15s trunk passed with JDK v1.8.0_72
          +1 compile 9m 53s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 25s trunk passed
          +1 mvnsite 2m 44s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 4m 47s trunk passed
          +1 javadoc 3m 0s trunk passed with JDK v1.8.0_72
          +1 javadoc 4m 10s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 4s the patch passed
          +1 compile 11m 10s the patch passed with JDK v1.8.0_72
          -1 javac 13m 34s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 739 unchanged - 0 fixed = 740 total (was 739)
          +1 javac 11m 10s the patch passed
          +1 compile 9m 57s the patch passed with JDK v1.7.0_95
          -1 javac 23m 31s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 735 unchanged - 0 fixed = 736 total (was 735)
          +1 javac 9m 57s the patch passed
          +1 checkstyle 1m 26s the patch passed
          +1 mvnsite 2m 40s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 19s the patch passed
          +1 javadoc 3m 2s the patch passed with JDK v1.8.0_72
          +1 javadoc 4m 12s the patch passed with JDK v1.7.0_95
          +1 unit 11m 10s hadoop-common in the patch passed with JDK v1.8.0_72.
          -1 unit 86m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 10m 1s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 80m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 34s Patch does not generate ASF License warnings.
          280m 32s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.cli.TestHDFSCLI
          JDK v1.7.0_95 Failed junit tests hadoop.ipc.TestRPC
            hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.ha.TestHAAppend



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787312/HDFS-9780.005.patch
          JIRA Issue HDFS-9780
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3ae828528173 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c3641ed
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          javac root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_72.txt
          javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14442/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 76MB
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14442/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 10s Maven dependency ordering for branch +1 mvninstall 9m 23s trunk passed +1 compile 11m 15s trunk passed with JDK v1.8.0_72 +1 compile 9m 53s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 25s trunk passed +1 mvnsite 2m 44s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 4m 47s trunk passed +1 javadoc 3m 0s trunk passed with JDK v1.8.0_72 +1 javadoc 4m 10s trunk passed with JDK v1.7.0_95 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 4s the patch passed +1 compile 11m 10s the patch passed with JDK v1.8.0_72 -1 javac 13m 34s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 739 unchanged - 0 fixed = 740 total (was 739) +1 javac 11m 10s the patch passed +1 compile 9m 57s the patch passed with JDK v1.7.0_95 -1 javac 23m 31s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 735 unchanged - 0 fixed = 736 total (was 735) +1 javac 9m 57s the patch passed +1 checkstyle 1m 26s the patch passed +1 mvnsite 2m 40s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 19s the patch passed +1 javadoc 3m 2s the patch passed with JDK v1.8.0_72 +1 javadoc 4m 12s the patch passed with JDK v1.7.0_95 +1 unit 11m 10s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 86m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 10m 1s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 80m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 34s Patch does not generate ASF License warnings. 280m 32s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.security.TestDelegationTokenForProxyUser JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.cli.TestHDFSCLI JDK v1.7.0_95 Failed junit tests hadoop.ipc.TestRPC   hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787312/HDFS-9780.005.patch JIRA Issue HDFS-9780 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3ae828528173 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c3641ed Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 javac root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_72.txt javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14442/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14442/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 76MB Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14442/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Patch looks faithful, no other comments from my end. Nice work Daniel.

          Show
          andrew.wang Andrew Wang added a comment - Patch looks faithful, no other comments from my end. Nice work Daniel.
          Hide
          templedf Daniel Templeton added a comment -

          Bumping Jenkins

          Show
          templedf Daniel Templeton added a comment - Bumping Jenkins
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 13s Docker failed to build yetus/hadoop:0ca8df7.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787373/HDFS-9780.006.patch
          JIRA Issue HDFS-9780
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14448/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 13s Docker failed to build yetus/hadoop:0ca8df7. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787373/HDFS-9780.006.patch JIRA Issue HDFS-9780 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14448/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          +1 LGTM.

          Any other comments Andrew Wang or Karthik Kambatla?

          Show
          rkanter Robert Kanter added a comment - +1 LGTM. Any other comments Andrew Wang or Karthik Kambatla ?
          Hide
          templedf Daniel Templeton added a comment -

          Good point, Robert Kanter. Fixed in this patch.

          Show
          templedf Daniel Templeton added a comment - Good point, Robert Kanter . Fixed in this patch.
          Hide
          rkanter Robert Kanter added a comment -

          Robert Kanter, what are your feelings on using the static variables to pass in objects instead of jumping through the hoops required to make the sink able to find its own handles to those objects?

          If it's too complicated, then that's fine.

          In some of the new tests, we have initKDC(), followed by a few statements, including createSecureConfig(), which can throw an Exception. The call to shutdownKdc() might not happen if createSecureConfig() throws an Exception because of where the try-finally is.

          Show
          rkanter Robert Kanter added a comment - Robert Kanter, what are your feelings on using the static variables to pass in objects instead of jumping through the hoops required to make the sink able to find its own handles to those objects? If it's too complicated, then that's fine. In some of the new tests, we have initKDC() , followed by a few statements, including createSecureConfig() , which can throw an Exception . The call to shutdownKdc() might not happen if createSecureConfig() throws an Exception because of where the try-finally is.
          Hide
          templedf Daniel Templeton added a comment -

          Oh, BTW, this patch adds a couple of long lines. I didn't think it was worth the hit to readability to break them up.

          Show
          templedf Daniel Templeton added a comment - Oh, BTW, this patch adds a couple of long lines. I didn't think it was worth the hit to readability to break them up.
          Hide
          templedf Daniel Templeton added a comment -

          Now that HDFS-9637 is in, here's a patch that will apply. It also takes care of Karthik Kambatla's comment about overzealous property checking.

          Show
          templedf Daniel Templeton added a comment - Now that HDFS-9637 is in, here's a patch that will apply. It also takes care of Karthik Kambatla 's comment about overzealous property checking.
          Hide
          kasha Karthik Kambatla added a comment -

          Quickly skimmed through the patch.

          Do we want to checkForProperty even if security is not enabled?

              // Validate config so that we don't get an NPE
              checkForProperty(conf, KEYTAB_PROPERTY_KEY);
              checkForProperty(conf, USERNAME_PROPERTY_KEY);
          

          Once the patch is updated based on HDFS-9637, will take a closer look.

          Show
          kasha Karthik Kambatla added a comment - Quickly skimmed through the patch. Do we want to checkForProperty even if security is not enabled? // Validate config so that we don't get an NPE checkForProperty(conf, KEYTAB_PROPERTY_KEY); checkForProperty(conf, USERNAME_PROPERTY_KEY); Once the patch is updated based on HDFS-9637 , will take a closer look.
          Hide
          kasha Karthik Kambatla added a comment -

          Canceling patch until HDFS-9637 gets committed.

          Show
          kasha Karthik Kambatla added a comment - Canceling patch until HDFS-9637 gets committed.
          Hide
          templedf Daniel Templeton added a comment -

          HDFS-9782 has been created to make the interval configurable.

          Show
          templedf Daniel Templeton added a comment - HDFS-9782 has been created to make the interval configurable.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HDFS-9780 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787008/HDFS-9780.004.patch
          JIRA Issue HDFS-9780
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14435/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HDFS-9780 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787008/HDFS-9780.004.patch JIRA Issue HDFS-9780 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14435/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          New patch incorporates tests. Now dependent on HDFS-9637.

          Show
          templedf Daniel Templeton added a comment - New patch incorporates tests. Now dependent on HDFS-9637 .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 39s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 33s trunk passed
          +1 compile 5m 57s trunk passed with JDK v1.8.0_72
          +1 compile 6m 36s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 32s trunk passed
          +1 javadoc 0m 52s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 4s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 40s the patch passed
          +1 compile 5m 52s the patch passed with JDK v1.8.0_72
          +1 javac 5m 52s the patch passed
          +1 compile 6m 39s the patch passed with JDK v1.7.0_95
          +1 javac 6m 39s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 50s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 2s the patch passed with JDK v1.7.0_95
          +1 unit 6m 38s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 6m 54s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          69m 26s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786710/HADOOP-12775.003.patch
          JIRA Issue HDFS-9780
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7c2e8167e799 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 193d27d
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14429/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14429/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 11m 39s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 33s trunk passed +1 compile 5m 57s trunk passed with JDK v1.8.0_72 +1 compile 6m 36s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 52s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 4s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 40s the patch passed +1 compile 5m 52s the patch passed with JDK v1.8.0_72 +1 javac 5m 52s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_95 +1 javac 6m 39s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 50s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 2s the patch passed with JDK v1.7.0_95 +1 unit 6m 38s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 6m 54s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 69m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786710/HADOOP-12775.003.patch JIRA Issue HDFS-9780 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7c2e8167e799 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 193d27d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14429/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14429/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          The sink is running under the test user (i.e. me) whereas the file system only knows about the two users I create: hdfs and sink. That would seem the issue I was having. I now have a different issue. When I use the current user instead of creating the sink user I get an even less informative GSS exception saying it can't setup the connection for my user. I have validated that the principals are the same on both sides.

          This is clearly some test setup oddity, because it works fine in a real cluster. Rather than digging all the way to the bottom of this issue here, I'd rather stick with the static variable to pass in an fs instance that works.

          Robert Kanter, what are your feelings on using the static variables to pass in objects instead of jumping through the hoops required to make the sink able to find its own handles to those objects?

          I think we should take the interval setting to a new JIRA. I'll file one shortly.

          Show
          templedf Daniel Templeton added a comment - The sink is running under the test user (i.e. me) whereas the file system only knows about the two users I create: hdfs and sink. That would seem the issue I was having. I now have a different issue. When I use the current user instead of creating the sink user I get an even less informative GSS exception saying it can't setup the connection for my user. I have validated that the principals are the same on both sides. This is clearly some test setup oddity, because it works fine in a real cluster. Rather than digging all the way to the bottom of this issue here, I'd rather stick with the static variable to pass in an fs instance that works. Robert Kanter , what are your feelings on using the static variables to pass in objects instead of jumping through the hoops required to make the sink able to find its own handles to those objects? I think we should take the interval setting to a new JIRA. I'll file one shortly.
          Hide
          rkanter Robert Kanter added a comment -

          I'm not sure about the checksum error itself, but how did you setup the secure mini cluster? Did you use MiniKDC?

          Show
          rkanter Robert Kanter added a comment - I'm not sure about the checksum error itself, but how did you setup the secure mini cluster? Did you use MiniKDC?
          Hide
          andrew.wang Andrew Wang added a comment -

          JIRA plan SGTM. We use milliseconds for most intervals, so recommend sticking with that. No idea about the filesystem, but if it's only when using a secure minicluster, it's probably some UGI issue. Robert Kanter might know more about such things.

          Show
          andrew.wang Andrew Wang added a comment - JIRA plan SGTM. We use milliseconds for most intervals, so recommend sticking with that. No idea about the filesystem, but if it's only when using a secure minicluster, it's probably some UGI issue. Robert Kanter might know more about such things.
          Hide
          templedf Daniel Templeton added a comment -

          Mind if I make the configurable interval a separate JIRA?

          Show
          templedf Daniel Templeton added a comment - Mind if I make the configurable interval a separate JIRA?
          Hide
          templedf Daniel Templeton added a comment -

          How about this on the tests... I'll take HDFS-9637 back to just covering HADOOP-12702 and HADOOP-12759, and then I'll move this JIRA over to HDFS and add the tests. Sound good?

          Show
          templedf Daniel Templeton added a comment - How about this on the tests... I'll take HDFS-9637 back to just covering HADOOP-12702 and HADOOP-12759 , and then I'll move this JIRA over to HDFS and add the tests. Sound good?
          Hide
          templedf Daniel Templeton added a comment -

          Thanks, Andrew Wang. The metrics system takes a class name as the source/sink to start. It's then instantiated in complete isolation. That works fine in reality, but it's ugly for testing. The alternative to the static suppliedConf is to have the test write out a configuration file and then convince the metrics system to read it. That's also pretty ugly.

          On the suppliedFilesystem variable, I still don't exactly understand why I need it. When using a secure mini-cluster, if I try to use FileSystem.get(), all operations fail with a checksum error. Know anything about that?

          I have thought about making the interval configurable. And now that you've asked the question, I guess I gotta. Would number of minutes be a reasonable way to configure the rollover interval?

          Show
          templedf Daniel Templeton added a comment - Thanks, Andrew Wang . The metrics system takes a class name as the source/sink to start. It's then instantiated in complete isolation. That works fine in reality, but it's ugly for testing. The alternative to the static suppliedConf is to have the test write out a configuration file and then convince the metrics system to read it. That's also pretty ugly. On the suppliedFilesystem variable, I still don't exactly understand why I need it. When using a secure mini-cluster, if I try to use FileSystem.get() , all operations fail with a checksum error. Know anything about that? I have thought about making the interval configurable. And now that you've asked the question, I guess I gotta. Would number of minutes be a reasonable way to configure the rollover interval?
          Hide
          andrew.wang Andrew Wang added a comment -

          This looks good overall, I applied HDFS-9637 to see the tests too. I'd like it if we posted tests and code together, since when it's split we don't have precommit coverage.

          Q about the testing static variables. I haven't worked much with the metrics system, but is there really no way to provide a configuration when instantiating? Needing statics to set these members is pretty gross, and makes the code bigger.

          Also, have you considered making the roll interval configurable? That way we wouldn't need a flushQuickly property. Seems useful since not all deployments will necessarily want hourly rolling.

          Show
          andrew.wang Andrew Wang added a comment - This looks good overall, I applied HDFS-9637 to see the tests too. I'd like it if we posted tests and code together, since when it's split we don't have precommit coverage. Q about the testing static variables. I haven't worked much with the metrics system, but is there really no way to provide a configuration when instantiating? Needing statics to set these members is pretty gross, and makes the code bigger. Also, have you considered making the roll interval configurable? That way we wouldn't need a flushQuickly property. Seems useful since not all deployments will necessarily want hourly rolling.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 8m 40s trunk passed
          +1 compile 9m 58s trunk passed with JDK v1.8.0_66
          +1 compile 9m 6s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 10s trunk passed
          +1 javadoc 1m 12s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 21s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 9m 56s the patch passed with JDK v1.8.0_66
          +1 javac 9m 56s the patch passed
          +1 compile 9m 27s the patch passed with JDK v1.7.0_91
          +1 javac 9m 27s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 23s the patch passed
          +1 javadoc 1m 17s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 23s the patch passed with JDK v1.7.0_91
          +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 14s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          81m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786710/HADOOP-12775.003.patch
          JIRA Issue HADOOP-12775
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 23a375ed5aca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5b59a0e
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8566/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8566/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 8m 40s trunk passed +1 compile 9m 58s trunk passed with JDK v1.8.0_66 +1 compile 9m 6s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 22s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 1m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 21s trunk passed with JDK v1.7.0_91 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 9m 56s the patch passed with JDK v1.8.0_66 +1 javac 9m 56s the patch passed +1 compile 9m 27s the patch passed with JDK v1.7.0_91 +1 javac 9m 27s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 23s the patch passed +1 javadoc 1m 17s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 23s the patch passed with JDK v1.7.0_91 +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 14s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 81m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786710/HADOOP-12775.003.patch JIRA Issue HADOOP-12775 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 23a375ed5aca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5b59a0e Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8566/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8566/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Rebased patch.

          Show
          templedf Daniel Templeton added a comment - Rebased patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HADOOP-12775 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786673/HADOOP-12775.002.patch
          JIRA Issue HADOOP-12775
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8563/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HADOOP-12775 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786673/HADOOP-12775.002.patch JIRA Issue HADOOP-12775 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8563/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HADOOP-12775 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786673/HADOOP-12775.002.patch
          JIRA Issue HADOOP-12775
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8561/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-12775 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786673/HADOOP-12775.002.patch JIRA Issue HADOOP-12775 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8561/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          This patch adds missing docs and cleans up a little. It will still fail to integrate until HADOOP-12759 is in.

          Show
          templedf Daniel Templeton added a comment - This patch adds missing docs and cleans up a little. It will still fail to integrate until HADOOP-12759 is in.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HADOOP-12775 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786595/HADOOP-12775.001.patch
          JIRA Issue HADOOP-12775
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8556/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-12775 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786595/HADOOP-12775.001.patch JIRA Issue HADOOP-12775 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8556/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          New tests will be added via HDFS-9637.

          Show
          templedf Daniel Templeton added a comment - New tests will be added via HDFS-9637 .
          Hide
          templedf Daniel Templeton added a comment -

          This patch may fail on integration pending HADOOP-12759.

          Show
          templedf Daniel Templeton added a comment - This patch may fail on integration pending HADOOP-12759 .

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              templedf Daniel Templeton
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development