Arpit Agarwal added a comment - 16/Oct/15 05:28 v1 patch. Documents missing settings in hdfs-default.xml. Add more detail to SecureMode.html, rewrite some sections. Document JournalNode settings. Update HdfsMultihoming.html to document new security settings added by HADOOP-12437 . Minor cleanup of HttpAuthentication.html to convert textual description to a table. I think this doc needs more detail but I don't understand this part of the configuration well enough to add content.

Hadoop QA added a comment - 16/Oct/15 07:12 -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 22s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 39s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 site 3m 32s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 44s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 common tests 7m 27s Tests passed in hadoop-common. -1 hdfs tests 50m 51s Tests failed in hadoop-hdfs.     102m 41s   Reason Tests Failed unit tests hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766986/HDFS-9254.01.patch Optional Tests site javadoc javac unit git revision trunk / cf23f2c hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13025/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13025/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13025/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13025/console This message was automatically generated.

Arpit Agarwal added a comment - 16/Oct/15 15:39 Documentation-only patch, needs no new tests. Test failures are unrelated to the patch.

Steve Loughran added a comment - 16/Oct/15 16:48 I'd thought "underdocumented" is a complete summary of kerberos info —good to see you trying to fix this. in the patch, you say that `dn@REALM.TLD` is allowed. I recall seeing some JIRAs where people were saying you get a stack trace unless you have the /HOST value of some kind or other

Arpit Agarwal added a comment - 16/Oct/15 18:43 The test failures do seem to be caused by my patch, oddly. I'll take a look. in the patch, you say that `dn@REALM.TLD` is allowed. I recall seeing some JIRAs where people were saying you get a stack trace unless you have the /HOST value of some kind or other Let me verify that in a test cluster, thanks for looking at the patch.

Arpit Agarwal added a comment - 23/Oct/15 16:30 - edited So yes it looks like at least the SaslRpcClient doesn't like principals without a host component. 192.168.56.80:8485: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Kerberos principal name does NOT have the expected hostname part: jn@EXAMPLE.COM; Host Details : local host is: "cm0.example.com/192.168.56.80" ; destination host is: "cm0.example.com" :8485; at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81) at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:232) at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:899) Whereas SecurityUtil handles them fine. We should be consistent. I'll file a separate bug to fix the SaslRpcClient , and any other components I run into, but also update the doc patch for now. Thanks for the catch.

Arpit Agarwal added a comment - 30/Oct/15 21:30 v3 patch. I had fixed incorrect defaults in hdfs-defaults.xml which broke some secure mode tests. Fixed those tests to initialize their configuration correctly. Remove mention of ServiceName@REALM.TLD principals.

Hadoop QA added a comment - 31/Oct/15 03:53 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 13s trunk passed +1 compile 4m 23s trunk passed with JDK v1.8.0_60 +1 compile 4m 6s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 55s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 2m 0s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 53s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 15s the patch passed +1 compile 4m 20s the patch passed with JDK v1.8.0_60 +1 javac 4m 20s the patch passed +1 compile 4m 17s the patch passed with JDK v1.7.0_79 +1 javac 4m 17s the patch passed +1 checkstyle 0m 59s the patch passed +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 27s the patch passed -1 whitespace 0m 0s The patch has 34 line(s) with tabs. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 48s the patch passed +1 javadoc 1m 58s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 48s the patch passed with JDK v1.7.0_79 -1 unit 6m 4s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 62m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 6m 49s hadoop-common in the patch passed with JDK v1.7.0_79. -1 unit 59m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 24s Patch generated 58 ASF License warnings. 181m 30s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.security.TestShellBasedIdMapping   hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.TestFileCreation   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.datanode.TestHSync   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.fs.permission.TestStickyBit   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDecommission   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.server.datanode.TestDiskError   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestReadWhileWriting   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.TestQuota   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestInjectionForSimulatedStorage JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.server.namenode.TestSnapshotPathINodes   hadoop.hdfs.server.datanode.TestDeleteBlockPool   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.TestFileCreation   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.datanode.TestHSync   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDecommission   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.server.datanode.TestDiskError   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestInjectionForSimulatedStorage JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769874/HDFS-9254.03.patch JIRA Issue HDFS-9254 Optional Tests asflicense site mvnsite javac javadoc mvninstall unit xml findbugs checkstyle compile uname Linux eb9252cb1415 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / b24fe06 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13309/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13309/console This message was automatically generated.

Steve Loughran added a comment - 31/Oct/15 11:44 check the indentation on some of the XML examples doesn't my little ebook merit a reference? you can paste in it's references too, or at least a subset of them: https://github.com/steveloughran/kerberos_and_hadoop/blob/master/sections/biblography.md

Arpit Agarwal added a comment - 13/Nov/15 22:32 Thanks for the review. I've added a references section including your book and fixed the tabs issue in hdfs-default.xml. I referred to your book when I was stuck debugging SPNEGO auth issues recently but didn't find an answer that helped me. Finally tracked it down to DNS-related quirks. It prompted me to document the parts I understood but I still don't understand SPNEGO well enough to improve its documentation. I don't envy the next person who has to debug Hadoop security quirks.

Hadoop QA added a comment - 14/Nov/15 02:24 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s docker + precommit patch detected. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 15s trunk passed +1 compile 5m 19s trunk passed with JDK v1.8.0_66 +1 compile 5m 33s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 58s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 5m 4s trunk passed +1 javadoc 2m 52s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 20s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 23s the patch passed +1 compile 5m 14s the patch passed with JDK v1.8.0_66 +1 javac 5m 14s the patch passed +1 compile 4m 55s the patch passed with JDK v1.7.0_79 +1 javac 4m 55s the patch passed +1 checkstyle 1m 6s the patch passed +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 35s the patch passed +1 javadoc 2m 30s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 19s the patch passed with JDK v1.7.0_79 -1 unit 8m 44s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 78m 53s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 9m 13s hadoop-common in the patch passed with JDK v1.7.0_79. -1 unit 71m 3s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 22s Patch generated 56 ASF License warnings. 225m 29s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl   hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSClientRetries Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-13 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772315/HDFS-9254.04.patch JIRA Issue HDFS-9254 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle uname Linux 552f8f9cbc2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / f94d892 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13507/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13507/console This message was automatically generated.

Hadoop QA added a comment - 14/Nov/15 03:15 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 2m 59s trunk passed +1 compile 4m 23s trunk passed with JDK v1.8.0_60 +1 compile 4m 15s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 57s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 33s trunk passed +1 javadoc 2m 2s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 50s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 12s the patch passed +1 compile 4m 26s the patch passed with JDK v1.8.0_60 +1 javac 4m 26s the patch passed +1 compile 4m 12s the patch passed with JDK v1.7.0_79 +1 javac 4m 12s the patch passed +1 checkstyle 0m 56s the patch passed +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 49s the patch passed +1 javadoc 2m 1s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 54s the patch passed with JDK v1.7.0_79 -1 unit 16m 58s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 51m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 7m 40s hadoop-common in the patch failed with JDK v1.7.0_79. -1 unit 52m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 20s Patch generated 56 ASF License warnings. 176m 39s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_79 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.fs.TestLocalFsFCStatistics   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.ha.TestEditLogTailer Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-14 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772315/HDFS-9254.04.patch JIRA Issue HADOOP-12570 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle uname Linux 42053c7cda06 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / f94d892 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/console This message was automatically generated.

Arpit Agarwal added a comment - 14/Nov/15 04:07 The failures look unrelated to the patch.

Arpit Agarwal added a comment - 16/Dec/15 18:34 Steve Loughran , any way I can convince you to review the latest patch? I think it addresses your feedback.

Steve Loughran added a comment - 16/Dec/15 19:42 yeah, find a way to make my ZK client code not to leave GSS API error 44 in the zk logs

Arpit Agarwal added a comment - 16/Dec/15 19:59 I have no idea what that's about. So I am going to see if I can find another reviewer.

Steve Loughran added a comment - 17/Dec/15 10:04 Some of the default values in tables are emphasised ; others converted to `code` font. I think they should all be `in code`. You can do `italiccode` for both line 271, /etc/security/keytab/nn.service.keytab should be ` /etc/security/keytab/nn.service.keytab` same for line 273 and /etc/security/keytab/spnego.service.keytab ... etc otherwise, +1

Chris Nauroth added a comment - 17/Dec/15 23:31 Nice improvements. Thank you, Arpit. In the references, the first entry can link to HADOOP-4487 , which is the original Hadoop security JIRA with design doc attached. In HdfsMultiHoming.md, the "Hostname Lookup" link isn't rendering correctly. There is a missing underscore. This would fix it: * [Hostname Lookup](#Hostname_Lookup) The descriptions in hdfs-default.xml are nice to have, but I'm concerned that they are not backwards-compatible. dfs.namenode.kerberos.internal.spnego.principal used to default to using the value of dfs.web.authentication.kerberos.principal . That would mean that if someone had configured dfs.web.authentication.kerberos.principal , but not dfs.namenode.kerberos.internal.spnego.principal , then it could have worked before this patch, but it would stop working after this patch. That probably partly explains the earlier SaslDataTransferTestCase failures. I think if tests need to be changed for this patch, then it's probably a sign that something has gone wrong.

Arpit Agarwal added a comment - 17/Dec/15 23:52 Hi Chris Nauroth , thanks for the review! The descriptions in hdfs-default.xml are nice to have, but I'm concerned that they are not backwards-compatible. There should be no incompatibility introduced by the patch. It just fixes wrong documentation to be in line with the code. e.g. dfs.namenode.kerberos.internal.spnego.principal has no default in code but it was documented as defaulting to dfs.web.authentication.kerberos.principal in hdfs-default.xml. From NameNodeHttpServer#start : HttpServer2.Builder builder = DFSUtil.httpServerTemplateForNNAndJN(conf, httpAddr, httpsAddr, "hdfs" , DFSConfigKeys.DFS_NAMENODE_KERBEROS_INTERNAL_SPNEGO_PRINCIPAL_KEY, DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY); A few unit tests broke after this patch because they were pulling hdfs-default.xml as a resource and depending on the incorrect defaults instead of initializing their own test configuration correctly. I'll post an updated patch addressing the remaining feedback.

Chris Nauroth added a comment - 18/Dec/15 00:01 Taking this example from the existing hdfs-default.xml: <property> <name>dfs.namenode.kerberos.internal.spnego.principal</name> <value>${dfs.web.authentication.kerberos.principal}</value> </property> If an hdfs-site.xml defined dfs.web.authentication.kerberos.principal , but did not define dfs.namenode.kerberos.internal.spnego.principal , then I think it still would have worked because of the token substitution logic in Configuration#substituteVars . In this patch, dfs.namenode.kerberos.internal.spnego.principal defaults to nothing, so the token substitution won't happen anymore. This is why I think there could be a compatibility problem.

Arpit Agarwal added a comment - 18/Dec/15 00:33 If an hdfs-site.xml defined dfs.web.authentication.kerberos.principal, but did not define dfs.namenode.kerberos.internal.spnego.principal, then I think it still would have worked because of the token substitution logic in Configuration#substituteVars. Nice catch Chris! So the reason this works is because hdfs-default.xml is available as a resource in the hadoop-hdfs jar. I'll update the next patch to restore the defaults.

Arpit Agarwal added a comment - 18/Dec/15 01:19 v05 patch addresses feedback from Steve: Made formatting of the default values consistent. A lot of the existing settings were inconsistently formatted so fixed those too, hence the increase in patch size. And Chris: Restored defaults to hdfs-default.xml and reverted test changes Fixed broken hyperlink in HdfsMultihoming.md, added a link to the HADOOP-4487 design doc. Also pretty-printed the markdown tables as they were getting messy to read.

Hadoop QA added a comment - 18/Dec/15 04:28 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 54s trunk passed +1 compile 8m 5s trunk passed with JDK v1.8.0_66 +1 compile 8m 40s trunk passed with JDK v1.7.0_91 +1 mvnsite 1m 55s trunk passed +1 mvneclipse 0m 28s trunk passed +1 javadoc 2m 2s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 48s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 33s the patch passed +1 compile 7m 59s the patch passed with JDK v1.8.0_66 +1 javac 7m 59s the patch passed +1 compile 8m 43s the patch passed with JDK v1.7.0_91 +1 javac 8m 43s the patch passed +1 mvnsite 1m 53s the patch passed +1 mvneclipse 0m 28s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 2m 1s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 51s the patch passed with JDK v1.7.0_91 +1 unit 7m 25s hadoop-common in the patch passed with JDK v1.8.0_66. -1 unit 51m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 7m 6s hadoop-common in the patch failed with JDK v1.7.0_91. -1 unit 49m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 175m 30s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits JDK v1.7.0_91 Failed junit tests hadoop.http.TestHttpServer   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778392/HADOOP-12570.05.patch JIRA Issue HADOOP-12570 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml uname Linux d8ecbdc05c6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6da9dee whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/console This message was automatically generated.

Chris Nauroth added a comment - 19/Dec/15 00:55 +1 for patch v05. I have committed this to trunk, branch-2 and branch-2.8. Arpit, thank you for the documentation improvements. Steve, thank you for reviewing.

Arpit Agarwal added a comment - 19/Dec/15 01:01 Thank you Steve and Chris.

Hudson added a comment - 19/Dec/15 01:20 FAILURE: Integrated in Hadoop-trunk-Commit #9002 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9002/ ) HADOOP-12570 . HDFS Secure Mode Documentation updates. Contributed by (cnauroth: rev 932956458dddabd9583b911c693386a56d723c0d) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsMultihoming.md hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md

Arpit Agarwal added a comment - 13/Jul/16 20:03 Cherry-picked this to 2.7.3.

Vinod Kumar Vavilapalli added a comment - 25/Aug/16 22:48 Closing the JIRA as part of 2.7.3 release.